Improve performance of H5Ovisit (and H5Ocopy, and functions that
retrieve and object name) by passing more information about the visited
object from the underlying H5G_visit routine to these callbacks.
Introduced an internal object callback for H5G_visit to facilitate this.
H5Ovisit1 and potentially H5Ovisit2 are still slow with deeply nested
groups due to the way these deprecated functions interact with the VOL
layer.
Various related changes, including refactoring part of dataset creation, and reworking how layout versions are calculated.
Needs more testing of filters that create very large chunks, but that will need code to turn it off in cases where we can't allocate >4GiB buffers. I tested this manually by increasing the expansion ration in the expand2 test in dsets, and it passed everything up until it tried to expand the datasets with H5Dset_extent() and my laptop ran out of disk space.
We should add code and testing to handle the case where the "size of size" is set less than 8 bytes. This is not a new issue, since it can be set to 2.
I will file issues for these, but I don't think they are necessary for the release.
In order to reduce hash collisions and take advantage of modern memory capacity, the default hash table size for the chunk cache has been increased from 521 to 8191. This means the hash table will consume approximately 64 KiB per open dataset. This value can be changed with H5Pset_cache() or H5Pset_chunk_cache(). This value was chosen because it is a prime number close to 8K.
* Change new chunk indexing methods to always encode chunk size as a 64
bit (size of lengths) integer, when using the 2.0 file format.
* Add CHANGELOG.md note
* Spelling
* Fix errors in parallel build
* Committing clang-format changes
* More parallel fixes.
* Committing clang-format changes
* Another parallel fix
* Fix parallel for real this time I hope
* Update function descriptions in dsets.c
* Fix spelling
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Change default file format to 1.8 across various tests and examples, updating file creation and access logic accordingly.
Behavior:
Default file format version changed to 1.8 in H5Pfapl.c.
Updated file creation and access to use 1.8 format in h5ex_g_compact.c and test_file_image.c.
Set earliest file format in multiple test files including cache_tagging.c, dtypes.c, and links.c.
Tests:
Modified expected output in tools/test/misc/expected/*.ls files to reflect new file format locations.
Adjusted test logic in test_file_image.c and cache_tagging.c to accommodate format changes.
Misc:
Added comments and TODOs for future format testing in test_file_image.c.
Minor variable renaming for clarity in test_file_image.c.
* Delay decoding VDS global heap block until the dataset is initialized,
so the layout in the ohdr does not have the VDS info. Also decode VDS
info before layout message copy.
* Add macro for searching for VDS source names in a hash table and adding
them if not found to reduce code duplication.
* H5FDset_fapl_ros3() sets page buffer size to 64MB by default. Added
H5F_PAGE_BUFFER_SIZE_DEFAULT macro indicating the page buffer size
setting is the default, and associated machinery and tests.
Add hash tables for detecting repeated source file and dataset names.
Share these repeated strings between mappings in memory. Add new
encoding format for VDS for shared names. Add tests and documentation
for these changes.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Matthew Larson <matthewjlar@gmail.com>
* Enable bitwise comparison of file image in file_image.c for all file
format versions. Add RELEASE.txt note for file image fix with latest
format (#1915).
* Fix error message in file image test.
* Fix error in H5Ddebug(). Add h5ls test cases.
* Committing clang-format changes
* Fix cmake (hopefully)
* Fix cmake again (hopefully)
* Fix issue with modification times showing up differently on different
systems. Add test case for v1 b-trees.
* Add cmake for new tests
* Add RELEASE.txt entry
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Properly clean up cache when failing to load an object header
* Don't check message type a second time in H5G__open_oid if the first attempt returns error
* Add more asserts to H5O__assert() to avoid segfaults
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Fix bug in array conversion with strided background buffer. Convert some
memmove calls to non-overlapping buffers to memcpy.
* Revert inappropriate use of mempy to memmove in H5T__conv_array
* Add testing
* Add RELEASE.txt note and overwrite test case.
This function allows the user to determine if the library performed selection I/O, vector I/O, or scalar (legacy) I/O during the last HDF5 operation performed with the provided DXPL. Expanded existing tests to check this functionality.
Initial implementation of selection I/O with type conversion. Allows
Parallel collective I/O with type conversion, as long as selection I/O
is enabled.
* Fix bug with cross platform compatibility of references within vlens.
No testing yet.
* Merge from multi_rd_wd_coll_io to a more recent branch from develop.
Untested, probably does not work yet.
* Committing clang-format changes
* Committing clang-format changes
* Fix many bugs in multi dataset branch. Mostly works, some issues in
SWMR tests.
* Committing clang-format changes
* Disable test in swmr.c that was failing due to bug in HDF5 unrelated to
multi dataset.
* Committing clang-format changes
* Fixed fortran multi-dataset tests
* Fixed xlf errors
* Added benchmark code for multi-datasets
* loops over datasets
* added missing error arg.
* Added gnuplot formatting
* Jonathan Kim original MD benchmarking code
* updated MD benchmarking code
* code clean-up
* Only make files in feature test mode
* misc clean-up
* removed TEST_MDSET_NO_LAST_DSET_2ND_PROC option
* Committing clang-format changes
* Change multi dataset API to use arrays of individual parameters instead
of the parameter struct.
* Committing clang-format changes
* Update to new multi dataset Fortran API and tests. (#1724)
* Update to new multi dataset Fortran API and tests.
* Sync Fortran with develop.
* skipping h5pget_mpio_actual_io_mode_f for now
* Fixed issue with dxpl_id, changed to variable size dim. (#1770)
* Remove "is_coll_broken" field from H5D_io_info_t struct
* Committing clang-format changes
* Minor cleanup in multi dataset code.
* Committing clang-format changes
* Clean up in multi dataset code.
* Committing clang-format changes
* Committing clang-format changes
* Fix speeling
* Fix bug in parallel compression. Switch base_maddr in io_info to be a
union.
* Committing clang-format changes
* Implement selection I/O support with multi dataset. Will be broken in
parallel until PR 1803 is merged to develop then the MDS branch.
* Committing clang-format changes
* Spelling
* Fix bug in multi dataset that could cause errors when only some of the
datasets in the multi dataset I/O used type conversion.
* Committing clang-format changes
* Integrate multi dataset APIs with VOL layer. Add async versions of
multi dataset APIs.
* Committing clang-format changes
* Spelling fixes
* Fix bug in non-parallel HDF5 compilation.
* Committing clang-format changes
* Fix potential memory/free list error. Minor performance fix. Other minor
changes.
* Committing clang-format changes
* Fix memory leak with memory dataspace for I/O.
* Committing clang-format changes
* Fix stack variables too large. Rename H5D_dset_info_t to
H5D_dset_io_info_t.
* Committing clang-format changes
* Remove mem_space_alloc field from H5D_dset_io_info_t. Each function is
now responsible for freeing any spaces it adds to dset_info.
* Committing clang-format changes
* fixed _multi Fortran declaration
* Refactor various things in (mostly) the serial I/O code path to make
things more maintainable.
* Committing clang-format changes
* updated to array based, doxygen, and examples
* Reinstate H5D_chunk_map_t, stored (via pointer) inside
H5D_dset_io_info_t.
* Change from calloc to malloc for H5D_dset_io_info_t and H5D_chunk_map_t.
Switch temporary dset_infos to be local stack variables.
* Committing clang-format changes
* format cleanup
* format cleanup
* added coll and ind
* Modify all parallel I/O paths to take dset_info instead of assuming
dset_info[0].
* Committing clang-format changes
* fixed output
* Rework parallel I/O code to work properly with multi dataset in more
cases. Fix bug in parallel compression.
* Committing clang-format changes
* Prevent H5D__multi_chunk_collective_io() from messing up collective opt
property for other datasets in I/O. Other minor cleanup. Add new test
case to t_pmulti_dset.c for H5FD_MPIO_INDIVIDUAL_IO, disabled for now
due to failures apparently unrelated to multi dataset code.
* Fix spelling
* Committing clang-format changes
* Replace N log N algorithm for finding chunk in
H5D__multi_chunk_collective_io() with O(N) algorithm, and remove use of
io_info->sel_pieces in that function.
* Committing clang-format changes
* Replace sel_pieces skiplist in io_info with flat array of pointers, use
qsort in I/O routine only when necessary.
* Committing clang-format changes
* Add new test case to mdset.c
* Committing clang-format changes
* Fix spelling
* Very minor fix in H5VL__native_dataset_read()
* Fix bug that could affect filtered parallel multi-dataset I/O.
* Add RM entries for H5Dread_multi(), H5Dread_multi_async(),
H5Dwrite_multi(), and H5Dwrite_multi_async()
* Unskip test in swmr.c
* Committing clang-format changes
* Eliminate H5D__pre_read and H5D__pre_write
* Remove examples/ph5mdsettest.c. Will fix and re-add as a test.
* Enable hyperslab combinations in mdset test
* Committing clang-format changes
* Clarify H5Dread/write_multi documentation.
* Fix bugs in multi-dataset I/O. Expand serial multi dataset test.
Update macro in parallel multi dataset test.
* Committing clang-format changes
* Spelling
* Remove obsolete entry in bin/trace
* Rework type conversion buffer allocation. Only one buffer is shared
between datasets in mdset mode, and it is malloced instead of calloced.
* Committing clang-format changes
* Fix bug in error handling in H5D__read/write
* added multi-dataset fortran check with optional dataset creation id (#2150)
* removed dup. dll entry
* Address comments from code review.
* Remove spurious changes in H5Fmpi.c
* Fix issue with reading unallocated datasets in multi-dataset mode.
Address other comments from code review.
* Committing clang-format changes
* Delay chunk index lookup from io_init to mdio_init so it doesn't add
overhead to single dataset I/O.
* Committing clang-format changes
* Fix inappropriate use of piece_count
* updated copyright on new file, removed benchmark from testing dir.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: M. Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>
* Fix bug in attribute type conversion where the background buffer would
not be initialized with the destination contents when necessary. Other
minor simplification.
* Committing clang-format changes
* Fix warnings.
* Address review comments.
* Add RELEASE.txt note for PR #2016
* Add GitHub number to release note
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Fix bug in attribute type conversion where the background buffer would
not be initialized with the destination contents when necessary. Other
minor simplification.
* Committing clang-format changes
* Fix warnings.
* Address review comments.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Fix HDFFV-11308 - dataset access properties disappear when using
H5Fstart_swmr_write.
* Committing clang-format changes
* Add test for H5Fstart_swmr_write() persisting DAPL settings. Fix bugs
in the library exposed by this test.
* Committing clang-format changes
* Replace accidental use of double_t in swmr.c with double.
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Fix HDFFV-11308 - dataset access properties disappear when using
H5Fstart_swmr_write.
* Committing clang-format changes
* Add test for H5Fstart_swmr_write() persisting DAPL settings. Fix bugs
in the library exposed by this test.
* Committing clang-format changes
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Added support for vector I/O calls to the VFD layer, and
associated test code. Note that this includes the optimization
to allow shortened sizes and types arrays to allow more space
efficient representations of vectors in which all entries are
of the same size and/or type. See the Selection I/o RFC for
further details.
Tested serial and parallel, debug and production on Charis.
serial and parallel debug only on Jelly.
* ran code formatter
quick serial build and test on jelly
* Add H5FD_read_selection() and H5FD_write_selection(). Currently only
translate to scalar calls. Fix const buf in H5FD_write_vector().
* Format source
* Fix comments
* Added vector read / write support to the MPIO VFD, with associated
test code (see testpar/t_vfd.c).
Note that this implementation does NOT support vector entries of
size greater than 2 GB. This must be repaired before release,
but it should be good enough for correctness testing.
As MPIO requires vector I/O requests to be sorted in increasing
address order, also added a vector sort utility in H5FDint.c This
function is tested in passing by the MPIO vector I/O extension.
In passing, repaired a bug in size / type vector extension management
in H5FD_read/write_vector()
Tested parallel debug and production on charis and Jelly.
* Ran source code formatter
* Minor optimization to vector I/O request sort routine.
Tested parallel / debug on Jelly.
* Add selection I/O to chunk code, used when: not using chunk cache, no
datatype conversion, no I/O filters, no page buffer, not using collective
I/O. Requires global variable H5_use_selection_io_g be set to TRUE.
Implemented selection to vector I/O transaltion at the file driver
layer.
* Fix formatting unrelated to previous change to stop github from
complaining.
* Add full API support for selection I/O. Add tests for this.
* Added "ctl" callback to the VFD interfance, and the associated
H5FDctl() and H5FD_ctl() calls.
Modified the MPIO VFD accordingly -- specifically:
Added ctl() call with op-code support to expose rank, size,
and communicator.
Modified H5FD_mpi_get_rank(), H5FD_mpi_get_size(),
and H5FD_mpi_get_comm() to use the new ctl() callback. In passing
removed the const qualifier from the file parameter of these
functions, as the file parameter of the ctl callback is not
const.
Deleted the old H5FD__mpio_mpi_rank(), H5FD__mpio_mpi_size(), and
H5FD__mpio_communicator() calls from the MPIO VFD.
Deleted H5FD_class_mpi_t from H5FDprivate.h, and modified the
MPIO VFD accordingly. Note that all VFDs now use H5FD_class_t,
with no special class for VFDs that that support MPI.
Some minor touch ups to the Neil's selection I/O mods in passing.
Tested serial and parallel, debug and production on charis and
jelly.
* Implement selection I/O for contiguous datasets. Fix bug in selection
I/O translation. Add const qualifiers to some internal selection I/O
routines to maintain const-correctness while avoiding memcpys.
* Add support for independent parallel I/O with selection I/O. Add
HDF5_USE_SELECTION_IO env var to control selection I/O (default off).
* Implement parallel collective support for selection I/O.
* Fix comments and run formatter.
* Committing clang-format changes
* Run formatter
* Update selection IO branch with develop (#1215)
Merged branch 'develop' into selection_io
* Sync with develop (#1262)
Updated the branch with develop changes.
* Committing clang-format changes
* fixed spelling (#1353)
* Implement big I/O support for vector I/O requests in the MPIO file
driver.
* Committing clang-format changes
* Update dummy file driver in h5test.c to match new struct.
* Fix another outdated H5FD_class_t struct.
* Free arrays in H5FD__mpio_read/write_vector() as soon as they're not
needed, to cut down on memory usage during I/O.
* Committing clang-format changes
* Address comments from code review. Fix const warnings with
H5S_SEL_ITER_INIT().
* Committing clang-format changes
* Fix warnings in selection I/O code
* Refactor H5FD__mpio_read/write_vector() to bring common code for
constructing MPI Datatype into a shared function.
* Committing clang-format changes
* Fix bugs in selection I/O code.
* Fix bugs in selection I/O code
* Address comments from review. Add "version" field to H5FD_class_t
struct and accompanying H5FD_CLASS_VERSION macro.
* Committing clang-format changes
* Committing clang-format changes
* Address comments from selection I/O PR
* Committing clang-format changes
* Fix spelling
* Fix bugs int test/vfd.c
Co-authored-by: mainzer <mainzer#hdfgroup.org>
Co-authored-by: jrmainzer <72230804+jrmainzer@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Scot Breitenfeld <brtnfld@hdfgroup.org>
Neil Fortner
