mirror of
https://github.com/HDFGroup/hdf5.git
synced 2026-09-25 04:09:44 +03:00
508695977838cff9f4f4f7ce3d5ff92d0ca765e2
433
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Larry Knox
|
5086959778 |
Sync develop changes April 4 -April 22 to hdf5_1_14 (#4437)
* Fixed dead links in docs (#4334) * Revert "Fixed dead links in docs (#4334)" (#4335) This reverts commit |
||
 Glenn Song
|
203cd53709 |
Update support branch (#4320)
Update support branch version number to 1.14.5-1 |
||
|
Larry Knox
|
3c86f0f0ce |
Sync develop changes March 20 - March 25 to hdf5_1_14. (#4241)
* Call memset before stat calls (#4202)
The buffers passed to stat-like calls are only partially filled in by
the call, leaving ununitialized memory areas when the stat buffers are
created on the stack.
This change memsets the buffers to 0 before the stat calls, quieting
the -fsanitze=memory complaints.
* Remove unused CMake configuration checks (#4199)
* Update link to Chunking in HDF5 page (#4203)
* Fix H5Pset_efile_prefix documentation error (#4206)
Fixes GH issue #1759
* Suggested header footer for NEWSLETTER (#4204)
* Suggested header footer for NEWSLETTER
* Updates
* Add NEWSLETTER.txt to h5vers script
* Fix grammar in README.md release template (#4201)
* Add back snapshot names (#4198)
* Use tar.gz extension for ABI reports (#4205)
* Fix issue with Subfiling VFD and multiple opens of same file (#4194)
* Fix issue with Subfiling VFD and multiple opens of same file
* Update H5_subfile_fid_to_context to return error value instead of ID
* Add helper routine to initialize open file mapping
* Reverts AC_SYS_LARGEFILE change (#4213)
We previously replaced local macros with AC_SYS_LARGEFILE, which is
unfortunately buggy on some systems and does not correctly set the
necessary defines, despite successfully detecting them.
This restores the previous macro hacks to acsite.m4
* Propagate group creation properties to intermediate groups (#4139)
* Add clarification for current behavior of H5Get_objtype_by_idx() (#4120)
* Addressed Fortran issues with promoted integers and reals via compilation flags (#4209)
* addressed issue wit promoted integers and reals
* added option to use mpi_f08
* Summarize the library version table (#4212)
Fixes GH-3773
* Fix URLs (#4210)
Also removed Copyright.html context because it's no longer valid.
* Fix 'make check-vfd' target for Autotools (#4211)
Changes Autotools testing to use HDF5_TEST_DRIVER environment
variable to avoid running tests that don't work well with several
VFDs
Restores old h5_get_vfd_fapl() testing function to setup a FAPL
with a particular VFD
Adds a macro for the default VFD name
* Revert "Addressed Fortran issues with promoted integers and reals via compil…" (#4220)
This reverts commit
|
||
|
Larry Knox
|
b69c6fcbf6 |
1 14 dev sync lrk (#4200)
* Added new H5E with tests. (#4049) Added Fortran H5E APIs: h5eregister_class_f, h5eunregister_class_f, h5ecreate_msg_f, h5eclose_msg_f h5eget_msg_f, h5epush_f, h5eget_num_f, h5ewalk_f, h5eget_class_name_f, h5eappend_stack_f, h5eget_current_stack_f, h5eset_current_stack_f, h5ecreate_stack_f, h5eclose_stack_f, h5epop_f, h5eprint_f (C h5eprint v2 signature) Addresses Issue #3987 * Don't load toolchain file in CMake workflows (#4077) * Add support for the new MSVC preprocessor (#4078) Microsoft has added a new, standards-conformant preprocessor to MSVC, which can be enabled with /Zc:preprocessor. This preprocessor trips over our HDopen() function-like variadic macro since it uses a hack that only works with the legacy MSVC preprocessor. This fix adds ifdefs to use the correct HDopen() macro depending on the MSVC preprocessor selected. Fixes #2515 * Increased H5FD_ROS3_MAX_SECRET_TOK_LEN to 4096 to accommodate long AWS secret tokens (#4064) ros3: increased H5FD_ROS3_MAX_SECRET_TOK_LEN to 4096; stratified the debugging statements so there is more control over the output * Close and reopen file during dset vlen IO API tests (#4050) - Close/reopen file and file objects to prevent cache from being used instead of actual I/O. - Moved vlen io test datasets under the dset container group instead of the root group - Moved the PASSED() invocation to after individual test cleanup in case an error occurs during H5Treclaim * New option for building with static CRT in Windows (#4062) * addressed compilation errors with gfortran 4.8 (#4082) * Fix bin/trace script w/ out params (#4074) The bin/trace script adds TRACE macros to public API calls in the main C library. This script had a parsing bug that caused functions that were annotated with /*out*/, etc. to be labeled as void pointers instead of typed pointers. This is mainly a developer feature and not visible to consumers of the public API. The bin/trace script now annotates public API calls properly. Fixes GH #3733 * Use H5T_STD_I32LE to create datatype in vds examples (#4070) Fixes issues when VDS examples are tested on BE systems * Remove printf debugging in H5HL code (#4086) * Fixed asserts due to H5Pset_est_link_info() values (#4081) * Fixed asserts due to H5Pset_est_link_info() values If large values for est_num_entries and/or est_name_len were passed to H5Pset_est_link_info(), the library would attempt to create an object header NIL message to reserve enough space to hold the links in compact form (i.e., concatenated), which could exceed allowable object header message size limits and trip asserts in the library. This bug only occurred when using the HDF5 1.8 file format or later and required the product of the two values to be ~64k more than the size of any links written to the group, which would cause the library to write out a too-large NIL spacer message to reserve the space for the unwritten links. The library now inspects the phase change values to see if the dataset is likely to be compact and checks the size to ensure any NIL spacer messages won't be larger than the library allows. Fixes GitHub #1632 * Fix copy-paste comments * update macOS support statement (#4084) * fixes compilation failures due to H5E additions (#4090) * Remove extra whitespaces from nvhpc-cmake action. (#4091) * Remove printf debugging in H5I package (#4088) * Add subfiling for h5dump filedriver option help message (#3878) * Merge HDF4 release workflow changes to hdf5 (#4093) * Update long double test with correct values (#4060) Update long double test with correct values * virtual creates must use the same datatype as the main file (#4092) * Fixed -Wdeprecated-copy-dtor warnings by implementing a copy assignment operator (#3306) Example warning was: warning: definition of implicit copy assignment operator for 'Group' is deprecated because it has a user-declared destructor [-Wdeprecated-copy-dtor] * Expand check for variable-length or reference types when clearing datatype conversion paths (#4085) When clearing out datatype conversion paths involving variable-length or reference datatypes on file close, also check for these datatypes inside compound or array datatypes * Remove H5B debug checks (#4089) The H5B (version 1 B-tree) package would add some computationally expensive integrity checks when H5B_DEBUG was defined. Due to their negative effects on performance, this option was rarely turned on, making the H5B__assert() check function stale, if not dead, code. This change: * Builds H5B__assert() when NDEBUG is not defined (the function relies on assert()) so it gets compiled more often. * Removes some printf debugging statements in the B-tree code * Removes all H5B "extra debug" checks that are leftover from past debugging sessions. Maintainers can add H5B__assert() selectively to perform integrity checks when debugging. * Removes the HDF5_ENABLE_DEBUG_H5B CMake option H5B_DEBUG now has no effect * Fix the last C++ stack size warning (#4099) * Clean up off_t usage (#4095) * Add comments to C++ and Fortran API calls that use off_t * Remove noise casts for small integers * Correct missing change of source path for S3 build (#4100) * Remove leading / from relative path. (#4101) * msvc: don't declare `HAVE_TIMEZONE` for older MSVC (#3956) It was introduced in MSVC 15 (Visual Studio 2017). * Remove a few H5O printf debugging statements (#4096) These were in H5Oint.c, were protected by H5O_DEBUG, and only dumped to stdout if the HDF5_DEBUG environment variable were set to do so. * Remove H5DEBUG() calls from H5Dmpio.c (#4103) Just use stdout when a stream is needed. * Remove printf debugging from H5Smpio.c (#4098) * Change how stats are printed in H5Z (#4097) H5Z used the soon-to-be-removed HDEBUG macro to decide if stats would be dumped and to what stream. This is now handled by a DUMP_DEBUG_STATS_g variable and the output is always sent to stdout. This is an internal change, not normally visible to users. * Refactor datatype conversion code to use pointers rather than IDs (#4104) The datatype conversion code previously used IDs for the source and destination datatypes rather than pointers to the internal structures for those datatypes. This was mostly due to the need for an ID for these datatypes that can be passed to an application-registered datatype conversion function or datatype conversion exception function. However, using IDs internally caused a lot of unnecessary ID lookups and hurt performance of datatype conversions in general. This was especially problematic for compound datatype conversions, where the ID lookups were occuring on every member of every compound element of a dataset. The code has now been refactored to use pointers internally and only create IDs for datatypes when necessary. Fixed a test issue in dt_arith where a library datatype conversion function was being cast to an application conversion function. Since the two have different prototypes, this started failing after the parameters for a library conversion function changed from hid_t to H5T_t * and an extra parameter was added. This appears to have worked coincidentally in the past since the only different between a library conversion function and application conversion function was an extra DXPL parameter at the end of an application conversion function Fixed an issue where memory wasn't being freed in the h5fc_chk_idx test program. Even though the program exits quickly after allocating the memory, it still causes failures when testing with -fsanitize=address * Minimize use of abort() (#4110) The abort() call is used at several places where it probably shouldn't. * Clean up a few things in H5T.c (#4105) * remove (size_t) noise casts * replace (hid_t)FAIL with H5I_INVALID_HID * Convert H5B__assert to use error checks (#4109) Switches assert() calls to HGOTO_ERROR in H5B__assert() so it can be used in production mode. Also renames it to H5B__verify_structure() to better reflect what it checks. * Move common variables out of cache test blocks (#4108) Fixes a stack size warning w/ XCode * Remove lint comments (#4107) * Change compression tests reference files to use masking for compression ratios (#4083) Rework TEST_FILTER tests to handle slightly different compression ratios * Add Doxygen for HDFS VFD (#4106) * Add Doxygen for HDFS VFD * Fix Doxygen warning * Update H5FDhdfs.h * long double tests has problems setting precision with offset (#4102) * long double tests has problems setting precision with offset * Disable long double until more development fixes issues * Fix up dsets test for some platforms with different long double format (#4114) * Adjust aocc workflow to fit the autotools/cmake pattern of other callable workflows. (#4115) * Implement ID creation optimization for container datatype conversions (#4113) Makes the datatype conversion context object available during both the initialization and conversion processes for a datatype conversion function, allowing the compound, variable-length and array datatype conversion functions to avoid creating IDs for the datatypes when they aren't necessary Adds internal H5CX_pushed routine to determine if an API context is available to retrieve values from Also adds error checking to several places in H5T.c and H5Tconv.c where the code had previously assumed object close operations would succeed * Handle IBM long double issues in dsets.c test_floattypes test (#4116) * Handle IBM long double issues in dsets.c test_floattypes test * Remove old cmake check (#4117) * Use AC_SYS_LARGEFILE on Autotools (#4119) We previously used a hack introduced in 1.8.5 to paper over differences in off_t and off64_t when determining the type sizes. We no longer explicitly support off64_t in the library and AC_SYS_LARGEFILE works fine. * Initialize selection type in chunk struct (#4087) * Overhaul CMake LFS support (#4122) Externally visible: * The HDF_ENABLE_LARGE_FILE option (advanced) has been removed * We no longer run a test program to determine if LFS works, which will help with cross-compiling * On Linux we now unilaterally set -D_LARGEFILE_SOURCE and -D_FILE_OFFSET_BITS=64, regardless of 32/64 bit system. CMake doesn't offer a nice equivalent to AC_SYS_LARGEFILE and since those options do nothing on 64-bit systems, this seems safe and covers all our bases. We don't set -D_LARGEFILE64_SOURCE since we don't use any of the POSIX 64-bit specific API calls like ftello64, as noted above. * We didn't test for LFS support on non-Linux platforms. We've added comments for how LFS should probably be supported on AIX and Solaris, which seem to be alive, though uncommon. PRs would be appreciated if anyone wishes to test this. Internal: * Drops off64_t size checks since this is unused (as in Autotools) * Remove HDF_EXTRA_FLAGS, which is now unused * Remove hack around deprecated LINUX_LFS Fixes #2395 * Update CMake comment about _POSIX_C_SOURCE (#4124) Was missng the 2008 pread/write info * Deprecate bin/cmakehdf5 (#4127) * Deprecate bin/cmakehdf5 * Add reference text * Don't set the rpath when linking statically (#4125) * Remove invalid compile flag (#4129) * Fix segfault in vlen io API test (#4130) * Update URLs in RELEASE.txt (#4132) * Add cygwin CI and update yaml files for consistency and accuracy (#4131) * Add cygwin CI * add cygwin packages * Correct option names * Cleanup yaml file and synch look and feel * Synch CI look and feel and correct path issues * Upgrade oneapi version * pwsh needs env: for vars * No continuation char for pwsh * restore correct pwsh step * Run subset of tests for cygwin workflow * Remove space chars in regex * restore full tests * Remove ros3 and hdfs VFDs from Autotools VFD list (#4142) These will never pass `make check` and would require a custom test suite for more comprehensive testing. * Skip part of dsets.c test for IBM long double type (#4136) * Capitalize option message for consistency. (#4141) * Fixed misc. H5E fortran failures due to previous PR (#4138) * fixed promotion of integers and reals tests and check-passthrough-vol failure * fixed cygwin issue * Fix Autotools -Werror cleanup (#4144) The Autotools temporarily scrub -Werror(=whatever) from CFLAGS, etc. so configure checks don't trip over warnings generated by configure check programs. The sed line originally only scrubbed -Werror but not -Werror=something, which would cause errors when the '=something' was left behind in CFLAGS. The sed line has been updated to handle -Werror=something lines. Fixes one issue raised in #3872 * Fix doxygen link to example function usage (#4133) * Remove useless headers (#4145) Removes unnecessary headers from C library source files. * Clean up some hbool_t/TRUE/FALSE stragglers (#4143) It looks like most of these snuck in via selection I/O work * Fix error when overwriting an indirectly nested vlen with a shorter sequence (#4140) * defined CMAKE_H5_HAVE_DARWIN (#4146) * Make the newsletter scheme work like HDF4 (#4149) * Remove at the end of list item. (#4151) * Fix buffer size calculation in the deflate filter (#4147) * Remove H5O header and friend status from H5A.c (#4150) * Remove HDF from Fortran 2003 configuration check message. (#4157) * Suppress H5Dmpio debugging output unless HDF5_DEBUG=d is set (#4155) * Header cleanup in C library (#4154) * Ensure H5FL header is included everywhere * Ensure H5SL header is included everywhere * Ensure H5MM header is included everywhere * Add Doxygen to H5FDmirror.h (#4158) * Remove lseek64 and stat64 symbols from CMake (#4163) We don't use these in the library. * Remove HAVE_IOEO checks from CMake (#4160) This was intended to check for thread-safety functionality on Windows. The required functionality has been standard since Windows Vista, so these checks can be removed. * Fix some minor warnings (#4165) * Bump the size of the mirror VFD IP field (#4167) The IP address string isn't big enought to hold an IPv4-mapped IPv6 address. * Fix mirror VFD script (#4170) This had directory problems when running locally. * Fix an issue where the Subfiling VFD's context cache grows too large (#4159) * Address code page issues w/ Windows file paths (#4172) On Windows, HDF5 attempted to convert file paths passed to open() and remove() to UTF-16 in order to handle Unicode file paths. This scheme does not work when the system uses code pages to handle non-ASCII file names. As suggested in the forum post below, we now also try to see if we can open the file with open(), which should handle systems where non-ASCII code pages are in use. https://forum.hdfgroup.org/t/open-create-hdf5-files-with-non-utf8-chars-such-as-shift-jis/11785 * Add Doxygen to API calls in H5VLnative.h (#4173) * Allow H5Soffset_simple to accept NULL offsets (#4152) The reference manual states that the offset parameter of H5Soffset_simple() can be set to NULL to reset the offset of a simple dataspace to 0. This has never been true, and passing NULL was regarded as an error. The library will now accept NULL for the offset parameter and will correctly set the offset to zero. Fixes HDFFV-9299 * Add filter plugin user guide text. Fix registered URL in docs (#4169) * Add support for _Float16 16-bit floating point type (#4065) Fixed some conversion issues with Clang due to problematic undefined behavior when casting a negative floating-point value to an integer Fixed a bug in the library's software integer to floating-point conversion function where a user's conversion exception function returning H5T_CONV_UNHANDLED in the case of overflows would result in incorrect data after conversion Added configure checks for functions and macros related to _Float16 usage since some compilers expose the datatype but not the functions or macros Fixed a dt_arith test failure when H5_WANT_DCONV_EXCEPTION isn't defined Fixed a few warnings from not explicitly casting some _Float16 variables upwards * Remove some H5T_copy calls that are now unnecessary (#4164) Removes some datatype copying calls that are now unnecessary after refactoring the datatype conversion code to use pointers internally rather than IDs Rewrites the enum conversion function so that it uses cached copies of the source and destination datatypes in order to avoid modifying the datatypes passed in Adds a 'recursive' field to the datatype conversion context which allows the conversion functions for members of a container datatype to skip unnecessary repetitive conversion setup code Changes internal datatype conversion callback functions so that the source and destination datatype structure pointers are const Removes some unused and unnecessary internal IDs registered with H5I_register * Add RELEASE.txt note for cmpd segfault fix (#4175) RELEASE notice for the fix in #3842 * Clean up CMake direct VFD handling (#4161) There's no need to build and run programs, or even check the operating system. We just need to check for O_DIRECT and posix_memalign(). * Capitalize linux for consistency (#4178) * Reworked H5Epush_f (#4153) * Add const to new _Float16 conversion routine parameters (#4181) * Update Release Specific Information link. (#4179) * Filter plugins updates for registration URL (#4180) * Update filter plugin URL to new location * Adjust test array size * Add daily VFD CI workflow (#4176) Adds testing of Subfiling VFD * Exclude shell tests from sanitizers (#4186) * Add a missing period at the end of sentence. (#4184) * last-file.txt should not be created for release workflow (#4185) * Skip part of dtypes.c _Float16 file size check for certain VFDs (#4182) * Fixes several MinGW + Autotools issues (#4190) * Fixes detection of various Windows libraries, etc. * Corrects alarm(2) configure checks * Uses Win32 threads by default w/ Pthreads override, if desired * Set _WIN32_WINNT correctly for MinGW * Fix setenv(3) wrapper for MinGW, which does not have getenv_s() MinGW Autotools support is still not Amazing, but this at least allows the library and tools build and is better about thread-safety * Add semicolons to the end of HSYS_GOTO_ERROR (#4193) Looks like we forgot these when we did the other macros. * Remove broken links (#4187) * Skip vlen IO API test for cache VOL (#4135) * Fix cache VOL segfault in vlen io test * Skip vlen IO API test * Handle certain empty subfiling environment variables (#4038) * h5diff compares attribute data like dataset data (#4191) Updates tools docs to indicate that dataset and attribute data are compared in the same way * A path component may include a dot with other characters (#4192) * Add RELEASE.txt note for recent datatype conversion improvements (#4195) * Add NEWSLETTER item about _Float16 support (#4197) * Correct download link for develop doxygen (#4196) * Update version in new .yml files. |
||
 Sean McBride
|
4aeaa9c2d4 | Issue #1824: Replaced most remaining sprintf with safer snprint (#4003) | ||
 Allen Byrne
|
fed6e095e5 | Add abi-complience check and upload to releases (#3996) | ||
|
Allen Byrne
|
56d9be146f |
Fix doxygen errors (#3962)
* Switch warnings as errors to default OFF * Enable mac docs * Add doxygen action uses step * Use html div around snippet * Allow preset name to be an argument to cmake-ctest.yml |
||
|
Allen Byrne
|
72e33ad772 | Merge examples and workflows from develop (#3918) | ||
 jhendersonHDF
|
fb84c696a4 |
Sync 1.14 branch with develop (#3887)
* Preserve MPI-I/O file hints when fapl is closed (#3755) * Fix for issue #3025: Save the MPI info in the file struct so H5Fget_access_plist() can retrieve it from there. * Add compression tests for subfiling (#3769) * Fix typo in comment (#3775) * Fixed a file handle leak in the core VFD (#3779) When opening a file with the core VFD and a file image, if the file already exists, the file check would leak the POSIX file handle. Fixes GitHub issue #635 * Fix a format string warning in the C++ examples (#3776) * Cancel running GitHub workflows on push to same PR (#3772) * Cancel running GitHub workflows on push to same PR * Remove github.sha from workflow concurrency groups * Print some messages in parallel tests on MPI rank 0 only (#3785) Avoids overly verbose output from all processes emitting progress, etc. info. * Avoid attempted use of NULL pointer in parallel compression code (#3786) The parallel compression test code tests for the case where all MPI ranks have no selection in a dataset when writing to it. Add an early exit to the code to avoid attempting to use a NULL pointer due to there being no work to do. * Don't install h5tools_test_utils test program on system (#3793) * Add Doxygen to H5FDsplitter.h (#3794) * H5FD_CURR_SPLITTER_VFD_CONFIG_VERSION * H5FD_SPLITTER_PATH_MAX * H5FD_SPLITTER_MAGIC * H5FD_splitter_vfd_config_t * H5Pset_fapl_splitter() * H5Pget_fapl_splitter() * Update Doxygen initializers & identifiers in VFDs (#3795) * Add Doxygen for all H5FD_<VFD> initializers * Add Doxygen for all H5FD_<VFD>_VALUE values * Mark H5FD_<vfd>_init() calls private in Doxygen * Fix memory corruption in 'MPI I/O FAPL preserve' test (#3806) * Fix usage of h5_clean_files in t_pflush2.c (#3807) * Fix parallel driver check in h5_fixname_real (#3808) * Fix a couple usages of MPI_Info_get (#3809) * Remove H5system.c warning on Windows oneAPI. (#3812) * Add processing of NVHPC flags in linux-gnulibc1 file (#3804) * Disable testing as tests are failing the same as in CMake * Use the current toolchain for examples as default (#3810) * Fix misc. warnings from GCC when compiling with -fsanitize=undefined (#3787) * Set NVHPC maximum optimization level to -O1 for now (#3800) * Set NVHPC maximum optimization level to -O1 for now Compiling HDF5 with NVHPC 23.5 - 23.9 results in test failures in 4 different test files that need to be resolved. Since those tests pass with an optimization level of -O1 (and -O0) and it is currently unclear whether the test failures are due to issues in HDF5 or issues in the 'nvc' compiler, set the maximum optimization level for NVHPC to -O1 until the test failures are resolved. * Disable nvhpc Java testing in CMake and amend known issues * Re-enable testing of Autotools nvhpc * Update some doxygen links to local refs (#3814) * Rework MPI Info FAPL preserve PR to use VFD 'ctl' operations (#3782) * Removed the use of C wrappers from H5P APIs. (#3824) * fix seg fault on frontier/cray * fix seg fault on frontier/cray * fix seg fault on frontier/cray * removed the use of h5pclose_c * removed the use of h5pclose_c * Fortran Wrappers H5VLnative_addr_to_token_f and H5VLnative_token_to_address_f (#3801) * Added H5VLnative_addr_to_token_f and H5VLnative_token_to_address_f * Added H5VLnative_addr_to_token_f and H5VLnative_token_to_address_f tests * Create test for H5Pget_dxpl_mpio (#3825) * Create test and add to testphdf5 * Renamed h5fuse.sh to h5fuse (#3834) * provide an alternative to mapfile for older bash * Disable FP exceptions in H5T init code (#3837) The H5T floating-point datatype initialization code can raise exceptions when handling signaling NaNs. This change disables FE_INVALID exceptions during initialization. Also removes the -ieee=full change for NAG Fortran as that shouldn't be necessary anymore. Fixes #3831 * Add intel oneapi windows build to CI CMake (#3836) * Remove printf format warning on Windows oneAPI. (#3838) * Correct ENV variables (#3841) * Remove Autotools sed hack (#3848) configure.ac contains a sed line that cleans up incorrect library flags which was added to paper over some bugs in earlier versions of the Autotools. These issues are not a problem with the current versions of the Autootols. The sed line causes problems on MacOS, so it has been removed. Fixes #3843 * Make filter unregister callbacks safe for VOL connectors (#3629) * Make filter callbacks use top-level API functions When using VOL connectors, H5I_iterate may not provide valid object pointers to its callback. This change keeps existing functionality in H5Zunregister() without using potentially unsafe pointers. * Filter callbacks use internal API * Skip MPI work on non-native VOL * Add extra space in comments for consistency (#3852) * Add extra space in comments for consistency * uncomment tfloatsattrs test * Update Actions badges to link to relevant workflow (#3850) * Add CMake long double cross-compile defaults (#3683) HDF5 performs a couple of checks at build time to see if long double values can be converted correctly (IBM's Power architecture uses a special format for long doubles). These checks were performed using TRY_RUN, which is a problem when cross-compiling. These checks now use default values appropriate for most non-Power systems when cross-compiling. The cache values can be pre-set if necessary, which will preempt both the TRY_RUN and the default. Affected values: H5_LDOUBLE_TO_LONG_SPECIAL (default no) H5_LONG_TO_LDOUBLE_SPECIAL (default no) H5_LDOUBLE_TO_LLONG_ACCURATE (default yes) H5_LLONG_TO_LDOUBLE_CORRECT (default yes) H5_DISABLE_SOME_LDOUBLE_CONV (default no) Fixes GitHub #3585 * Updates for building and testing VOL connectors * Fix issue with HDF5_VOL_ALLOW_EXTERNAL CMake variable * Initialize parallel testing with MPI_THREAD_MULTIPLE when testing API * Add CMake variable to allow specifying a VOL connector's package name * Remove call to MPI_Init in serial API tests While previously necessary, it now interferes with VOL connectors that may need to be initialized with MPI_THREAD_MULTIPLE * Fixes for CI and presets (#3853) * Change dest for doxygen (#3856) * Implement selection vector I/O with collective chunk filling (#3826) * Changes for ECP-344: Implement selection vector I/O with collective chunk filling. Also fix a bug in H5FD__mpio_write_vector() to account for fixed size optimization when computing max address. * Fixes based on PR review comments: For H5Dchunk.c: fix H5MM_xfree() For H5FDmpio.c: 1) Revert the fix to H5FD__mpio_write_vector() 2) Apply the patch from Neil on the proper length of s_sizes reported by H5FD__mpio_vector_build_types() * Put back the logic of dividing up the work among all the mpi ranks similar to the original H5D__chunk_collective_fill() routine. * Add a test to verify the fix for the illegal reference problem in H5FD__mpio_write_vector(). * Do not publish compression headers or docs (#3865) * Fix typo: look -> loop (#3866) * Moved the README to markdown and expanded its overview of the files, file generation, and other Fortran wrapper development practices as mentioned in the HDF5 architectural document. I added a new figure and included the SVG file and the original xfig file it was generated from. (#3862) * Add HDF5_DISABLE_TESTS_REGEX option to skip tests (#3859) * Fix typo in error message for `MPI_Type_dup`. (#3867) * Complete the `if command line option` sentence. (#3868) * Fix h5dump segmentation fault when --vfd-value option is used (#3873) * Updated URL in funding.yml (#3882) Using new shortened URL, might look better. * Remove unused variable from unmerged changes * Update src/H5Tinit_float.c |
||
|
Allen Byrne
|
528a8e7f46 |
Sync CMake changes from develop (#3811)
Guard test dirs with HDF5_EXTERNALLY_CONFIGURED Add toolchain support to examples testing |
||
|
Larry Knox
|
71d8975b54 |
Update version to 1.14.4-1. (#3799)
Clean 1.14.3 entries from RELEASE.txt. Add hdf5-1.14.3-RELEASE.txt contents to HISTORY-1_14.txt. |
||
|
jhendersonHDF
|
a91be87f07 |
Sync with develop (#3764)
* Add missing test files to distclean target (#3734) Cleans up new files in Autotools `make distclean` in the test directory * Add tools/libtest to Autotools builds (#3735) This was only added to CMake many years ago and tests the tools library. * Clean up onion VFD files in tools `make clean` (#3739) Cleans up h5dump and h5diff *.onion files in the Autotools when runing `make clean`. * Clean Java test files on Autotools (#3740) Removes generated HDF5 and text output files when running `make clean`. * Clean the flushrefresh test dir on Autotools (#3741) The flushrefresh_test directory was not being cleaned up w/ `make clean` under the Autotools * Fix file names in tfile.c (#3743) Some tests in tfile.c use h5_fileaccess to get a VFD-dependent file name but use the scheme from testhdf5, reusing the FILE1 and FILE8 names. This leads to files like test1.h5.h5 which are unintended and not cleaned up. This changes the filename scheme for a few tests to work with h5test, resulting in more informative names and allowing the files to be cleaned up at the end of the test. The test files have also been added to the `make clean` target for the Autotools. * Clean Autotools files in parallel tests (#3744) Adds missing files to `make clean` for parallel, including Fortran. * Add native VOL checks to deprecated functions (#3647) * Add native VOL checks to deprecated functions * Remove unneeded native VOL checks * Move native checks to top level calls * Fix buffer overflow in cache debugging code (#3691) * update stat arg for apple (#3726) * update stat arg for apple * use H5_HAVE_DARWIN for Apple ifdef * fix typo * removed duplicate H5_ih_info_t * added fortran async test to cmake * Fix windows cpack error in WiX package. (#3747) * Add a simple cache to the ros3 VFD (#3753) Adds a small cache of the first N bytes of a file opened with the read-only S3 (ros3) VFD, where N is 4kiB or the size of the file, whichever is smaller. This avoids a lot of small I/O operations on file open. Addresses GitHub issue #3381 * Update Autotools to correctly configure oneAPI (#3751) * Update Autotools to correctly configure oneAPI Splits the Intel config files under the Autotools into 'classic' Intel and oneAPI versions, fixing 'unsupported option' messages. Also turns off `-check uninit` (new in 2023) in Fortran, which kills the H5_buildiface program due to false positives. * Enable Fortran in oneAPI CI workflow * Turn on Fortran in CMake, update LD_LIBRARY_PATH * Go back to disabling Fortran w/ Intel For some reason there's a linking problem w/ Fortran error while loading shared libraries: libifport.so.5: cannot open shared object file: No such file or directory * Add h5pget_actual_selection_io_mode fortran wrapper (#3746) * added h5pget_actual_selection_io_mode_f test * added tests for h5pget_actual_selection_io_mode_f * fixed int_f type conversion * Update fortran action step (#3748) * Added missing DLL for H5PGET_ACTUAL_SELECTION_IO_MODE_F (#3760) * add missing H5PGET_ACTUAL_SELECTION_IO_MODE_F dll * Bump the ros3 VFD cache to 16 MiB (#3759) * Fix hangs during collective I/O with independent metadata writes (#3693) * Fix some issues with collective metadata reads for chunked datasets (#3716) Add functions/callbacks for explicit control over chunk index open/close Add functions/callbacks to check if chunk index is open or not so that it can be opened if necessary before temporarily disabling collective metadata reads in the library Add functions/callbacks for requesting loading of additional chunk index metadata beyond the chunk index itself * Fix failure in t_select_io_dset when run with more than 10 ranks (#3758) * Fix H5Pset_evict_on_close failing regardless of actual parallel use (#3761) Allow H5Pset_evict_on_close to be called regardless of whether a parallel build of HDF5 is being used Fail during file opens if H5Pset_evict_on_close has been set to true on the given File Access Property List and the size of the MPI communicator being used is greater than 1 |
||
  
|
4a028905e4 |
Sync 1.14 branch with develop (#3705)
* Address nagfor exceptions stoppage. (#3658) * added cmake ieee flag for nagfor * generalized determining the nag compiler * fixing some misc. NAG warnings * Simplify. (#3659) * Address @jhendersonHDF review * Add expedited testing support to t_filters_parallel (#3665) * Remove clang warnings (#3656) * Fixes test failure for gfortran -O2 and -O3, -fdefault-real-16 (#3662) * added cmake ieee flag for nagfor * fixes gfortran -O2 and -O3, -fdefault-real-16 * fixed sync * updated release notes * Fix link error on clang17/gfortran13/macOS-13 (#3666) (#3671) * Correct fortran CMake generator expressions (#3670) * Add AOCC GitHub Action (#3504) (#3657) * Fix uninitialized subfiling test variable (#3675) Picked up by gcc 10 on skybridge. Probably spurious, but no harm in initializing it to a "bad" value. * Add support for AOCC & Flang w/ the Autotools (#3674) * Adds a config/clang-fflags options file to support Flang * Corrects missing "-Wl," from linker options in the libtool wrappers when using Flang, the MPI Fortran compiler wrappers, and building the shared library. This would often result in unrecognized options like -soname. * Enable -nomp w/ Flang to avoid linking to the OpenMPI library. CMake can build the parallel, shared library w/ Fortran using AOCC and Flang, so no changes were needed for that build system. Fixes GitHub issues #3439, #1588, #366, #280 * Fix a strncpy call to use dest size not src (#3677) A strncpy call in a path construction call used the size of the src buffer instead of the dest buffer as the limit n. This was switched to use the dest size and properly terminate the string if truncation occurs. * Remove CANBE_UNUSED() from subfiling VFD (#3678) This macro was an attempt to quiet warnings about release mode unused variables that only appear in asserts. It resolves to a void cast, which doesn't quiet warnings when an assignment has already taken place. * Suppress MPI_Waitall warnings w/ MPICH (#3680) MPICH defines MPI_STATUSES_IGNORE (a pointer) to 1, which raises warnings w/ gcc. This is a known issue that the MPICH devs are not going to fix. See here: https://github.com/pmodels/mpich/issues/5687 This fix suppresses those issues w/ gcc * Fix a possible NULL pointer dereference in tests (#3676) The dtypes test could dereference a NULL pointer if a strdup call failed. * Fix printf warnings in t_mpi (#3679) * Fix printf warnings in t_mpi The type of MPI_Offset varies with implementation. In MPICH, it's long, which raises warnings when we attempt to use long long format specifiers. Casting to long long fixes the warnings. * Fix invalid memory access in S3 comms (#3681) In the ros3 VFD, passing an empty string parameter to an internal API call could result in accessing the -1th element of a string. This would cause failures on big-endian systems like s390x. This parameter is now checked before writing to the string. Fixes GitHub #1168 * Add Doxygen for H5Pset_fapl_sec2() (#3685) * * switch to using time function instead of date function (#3690) * Initialize API context MPI types to MPI_BYTE (#3688) * Add test info output to t_filters_parallel (#3696) * Suppress format string warnings in subfiling test (#3699) * Fix unused variable in tselect.c (#3701) * Fix unused variable warning in H5F_sfile_assert_num (#3700) * Restore floating-point suffixes in tests (#3698) A prior commit removed too many F suffixes. This restores the suffixes for float variables. * Sync with changes from develop --------- Co-authored-by: Scot Breitenfeld <brtnfld@hdfgroup.org> Co-authored-by: H. Joe Lee <hyoklee@hdfgroup.org> Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com> Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com> |
||
|
jhendersonHDF
|
ea3f926057 |
1.14 sync with develop (#3660)
* Rework tools.cmake and add C flags (#3110) * Fix gh pages so that the doxygen files are uploaded (#3102) * Add workspace path * add debug * Make one job so workspace files are available * Put doxygen docs under docs folder in gh-pages * Fix a misc warning in test/vol.c (#3112) The compiler complains about using integers instead of size_t for some sizes. * Remove H5detect and H5make_libsettings (#3104) Removes H5detect and H5make_libsettings from the build and replaces their functionality with things that don't affect cross-compiling. H5detect --> floating-point types are now detected on library load H5make_libsettings --> Moved functionality to a new H5build_settings.c template file * clang-tidy clang-analyzer-core issues addressed (#3113) src/H5system.c:1293:13: warning: Dereference of null pointer [clang-analyzer-core.NullDereference] src/H5trace.c:4048:17: warning: Passed-by-value struct argument contains uninitialized data (e.g., via the field chain: 'initial.user') [clang-analyzer-core.CallAndMessage] * Add note for issue 3056 (#3117) * Use 1.14 toolchain (#3116) * Remove the checkposix script (#3122) This script was used to ensure that all non-HDF5 calls were prefixed with 'HD'. We are removing this scheme so this script is no longer needed. * Remove unused HD macros (#3120) * Remove unused HD macros The library prefixes most C and POSIX API calls with 'HD'. We are going to start removing these so the code looks like normal C. This PR removes most of the unused HD markup macros. * Replace ntohl/ntohs * Adds an optional version arg to bin/format_source (#3119) * Clean up mirror VFD code in utils dir (#3121) * Remove dead code * Replace mybzero with memset * Replace hbool_t/TRUE/FALSE with bool/true/false * Fix spelling issues flagged by codespell (#3130) * Make autogen.sh output message consistent (#3128) * Add Python for HDF-EOS zoo description (#3129) * Fix function name in comment in ros3 VFD (#3131) * Revert long double checks (#3133) * Revert "Remove long double conversion work-arounds (#3097)" This reverts commit |
||
|
Dana Robinson
|
51ff48b887 | Fix several spelling/grammar issues (#3621) (#3632) | ||
|
Allen Byrne
|
cf6d3e0737 | Correct dimensions in java test (#3484) | ||
|
Larry Knox
|
3262891c6b |
Merges from develop Aug 2-4 (#3354)
* Fix CVE-2018-11202 (#3330) A malformed file could result in chunk index memory leaks. Under most conditions (i.e., when the --enable-using-memchecker option is NOT used), this would result in a small memory leak and and infinite loop and abort when shutting down the library. The infinite loop would be due to the "free list" package not being able to clear its resources so the library couldn't shut down. When the "using a memory checker" option is used, the free lists are disabled so there is just a memory leak with no abort on library shutdown. The chunk index resources are now correctly cleaned up when reading misparsed files and valgrind confirms no memory leaks. * Fix CVE-2018-13867 (#3336) * Windows runtime items go into the bin folder (#3320) |
||
|
Larry Knox
|
51715c1be6 | Set version to 1.14.3-1. (#3343) | ||
|
Dana Robinson
|
00f5d01c83 | Normalize java w/ develop (#3283) | ||
|
Allen Byrne
|
bfeb1b0dd5 |
Fix doxygen and actions - merge from develop (#3109)
* Update the workflows to changes in develop * Disable java for error only action * Fix Fortran doxygen * add fortran alias * Merge changes from develop to fix actions |
||
|
Larry Knox
|
722574e05c |
Update version (#2971)
* Update version Copy RELEASE.txt from 1.14.1-2 release into HISTORY-1_14.txt Clear out RELEASE.txt |
||
|
Dana Robinson
|
7707859279 | Merge with develop (#2790) | ||
|
Dana Robinson
|
69c720bffa |
Merge "Use additional CPUs" and related fixes (#2742)
* 2720 Fix java depends and pdb lib install * 2695 Use additional CPUs in GitHub actions |
||
|
Allen Byrne
|
2319d4a092 | Change name of test file to avoid conflict (#2703) | ||
|
Allen Byrne
|
097f65e7a6 | Update java logger version (#2644) | ||
|
Allen Byrne
|
d51c97e7f5 |
1.14 Merge doxygen plist tables changes #2470 from develop (#2503)
* Merge doxygen plist tables changes from develop * Doxy corrections |
||
|
Allen ByrneandDana Robinson
|
04bd873d77 |
1.14 Fix some doxygen warnings (#2323)
* Fix some doxygen warnings * Remove javadocs * Fix spelling * Fix section format * Correct doxygen function name * Remove duplicate Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com> |
||
|
Larry KnoxandAllen Byrne
|
23be40ca4f |
Hdf5 1 14 (#2296)
* Update versions for hdf5_1_14 develop branch after creation of hdf5_1-14_0 branch for 1.14.0 release. * Update windows worker compilers (#2286) * Update windows worker compilers * Update bin and test issues * Update script and revert java test * Update scripts and versions in java files. Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com> |
||
|
Larry Knox
|
4e5bdb8c4e |
Update versions for hdf5_1_14 develop branch after creation of (#2294)
hdf5_1-14_0 branch for 1.14.0 release. |
||
|
Allen Byrne
|
2376723d4a |
Develop jni trans (#2266)
* Add compound and refactor out atomic types * Add Array String tests back * Convert Attribute version of compound example * Update transfer atom8ic read to return object |
||
|
Allen Byrne
|
719d800499 | Refactor JNI translate functions to a recursive switch on datatype (#2232) | ||
|
Larry Knox
|
11dfa25910 |
Update copyright headers (#2184)
* Updated source file copyright headers to remove "Copyright by the Board of Trustees of the University of Illinois", which is kept in the top-level COPYING file. |
||
|
Larry Knox
|
0173706826 |
Update develop branch version to 1.13.4-1 (#2189)
* Updated version to 1.13.4-1 after creating hdf5_1_13_3 branch for 1.13.3 release. Added new version to exceptions in lines 74-5 of src/H5.c |
||
|
Allen Byrne
|
9c61f7b1e1 |
JNI: Improve varlen datatype handling in H5A/H5D read/write functions (#2156)
* Improve JNI VL datatype read/write to handle complex combinations * Implement VL of VL JNI writes and reads * Add Java VLofVL test for attributtes * Changes to address review issues * Fix H5Aread vl blocks |
||
|
Allen Byrne
|
ea902f9ace |
Implement option for using CMake GNUInstallDirs (#2175)
* Implement option for using CMake GNUInstallDirs * Add release notes |
||
|
Allen Byrne
|
c114bd64b0 | Fix mostly java warnings, and cpp cast (#2134) | ||
|
Allen Byrne
|
45178c87a3 | develop Merge doxygen from 1.12 branch (#2095) | ||
|
Allen Byrne
|
c8fdd92cd4 |
Update CMake code to 3.18 minimum version (#2080)
* Update CMake code to 3.18 minimum version * Updated release note |
||
|
jhendersonHDF
|
5719187a34 | Fix Java tests for Developer build modes (#2079) | ||
|
jhendersonHDF
|
281db5876e | Cleanup some warnings in Java JNI code (#2034) | ||
|
Dana Robinson
|
7f6261a677 |
Removes cmake_ext_mod (files moved to cmake) (#2041)
At one time this content was a git submodule shared between several projects but that time has long passed. |
||
|
Larry Knox
|
25ef534039 |
Update version to 1.13.3-1 after creating hdf5_1_13_2 branch for 1.13.2 (#2004)
release. Added new version to exceptions in line 75 of src/H5.c. |
||
|
Larry Knox
|
029cde6111 |
Revert "Increment version to 1.13.3-1 after creating hdf5_1_13_2 branch for release"
This reverts commit
|
||
|
Larry Knox
|
c58558e872 |
Increment version to 1.13.3-1 after creating hdf5_1_13_2 branch for release
of HDF5 1.13.2. |
||
|
Allen Byrne
|
ec5100bf26 | Fix Cmake depends prop (#1995) | ||
|
Allen Byrne
|
29c8937f60 | Remove misnamed java vol clear test (#1990) | ||
 luzpaz
|
174505ca3c |
Fix typos found in the rest of the hdf5 code-base (#1985)
* Fix typos found in the rest of the hdf5 code-base
* Typo in tool.cmake
* Revert "Typo in tool.cmake"
This reverts commit
|
||
|
Allen Byrne
|
956282fa6f | Develop cmake clean (#1971) | ||
|
Allen Byrne
|
ae414872f5 |
Develop clang 13 format (#1933)
* Update format source to clang 13 * More format changes |
||
     
|
27bb358f7a |
Subfiling VFD (#1883)
* Added support for vector I/O calls to the VFD layer, and
associated test code. Note that this includes the optimization
to allow shortened sizes and types arrays to allow more space
efficient representations of vectors in which all entries are
of the same size and/or type. See the Selection I/o RFC for
further details.
Tested serial and parallel, debug and production on Charis.
serial and parallel debug only on Jelly.
* ran code formatter
quick serial build and test on jelly
* Add H5FD_read_selection() and H5FD_write_selection(). Currently only
translate to scalar calls. Fix const buf in H5FD_write_vector().
* Format source
* Fix comments
* Add selection I/O to chunk code, used when: not using chunk cache, no
datatype conversion, no I/O filters, no page buffer, not using collective
I/O. Requires global variable H5_use_selection_io_g be set to TRUE.
Implemented selection to vector I/O transaltion at the file driver
layer.
* Fix formatting unrelated to previous change to stop github from
complaining.
* Add full API support for selection I/O. Add tests for this.
* Implement selection I/O for contiguous datasets. Fix bug in selection
I/O translation. Add const qualifiers to some internal selection I/O
routines to maintain const-correctness while avoiding memcpys.
* Added vector read / write support to the MPIO VFD, with associated
test code (see testpar/t_vfd.c).
Note that this implementation does NOT support vector entries of
size greater than 2 GB. This must be repaired before release,
but it should be good enough for correctness testing.
As MPIO requires vector I/O requests to be sorted in increasing
address order, also added a vector sort utility in H5FDint.c This
function is tested in passing by the MPIO vector I/O extension.
In passing, repaired a bug in size / type vector extension management
in H5FD_read/write_vector()
Tested parallel debug and production on charis and Jelly.
* Ran source code formatter
* Add support for independent parallel I/O with selection I/O. Add
HDF5_USE_SELECTION_IO env var to control selection I/O (default off).
* Implement parallel collective support for selection I/O.
* Fix comments and run formatter.
* Update selection IO branch with develop (#1215)
Merged branch 'develop' into selection_io
* Sync with develop (#1262)
Updated the branch with develop changes.
* Implement big I/O support for vector I/O requests in the MPIO file
driver.
* Free arrays in H5FD__mpio_read/write_vector() as soon as they're not
needed, to cut down on memory usage during I/O.
* Address comments from code review. Fix const warnings with
H5S_SEL_ITER_INIT().
* Committing clang-format changes
* Feature/subfiling (#1464)
* Initial checkin of merged sub-filing VFD.
Passes regression tests (debug/shared/paralle) on Jelly.
However, bugs and many compiler warnings remain -- not suitable
for merge to develop.
* Minor mods to src/H5FDsubfile_mpi.c to address errors reported by autogen.sh
* Code formatting run -- no test
* Merged my subfiling code fixes into the new selection_io_branch
* Forgot to add the FindMERCURY.cmake file. This will probably disappear soon
* attempting to make a more reliable subfile file open which doesn't return errors. For some unknown reason, the regular posix open will occasionally fail to create a subfile. Some better error handling for file close has been added.
* added NULL option for H5FD_subfiling_config_t in H5Pset_fapl_subfiling (#1034)
* NULL option automatically stacks IOC VFD for subfiling and returns a valid fapl.
* added doxygen subfiling APIs
* Various fixes which allow the IOR benchmark to run correctly
* Lots of updates including the packaging up of the mercury_util source files to enable easier builds for our Benchmarking
* Interim checkin of selection_io_with_subfiling_vfd branch
Moddified testpar/t_vfd.c to test the subfiling vfd with default configuration.
Must update this code to run with a variety of configurations -- most particularly
multiple IO concentrators, and stripe depth small enough to test the other IO
concentrators.
testpar/t_vfd.c exposed a large number of race condidtions -- symtoms included:
1) Crashes (usually seg faults)
2) Heap corruption
3) Stack corruption
4) Double frees of heap space
5) Hangs
6) Out of order execution of I/O requests / violations of POSIX semantics
7) Swapped write requests
Items 1 - 4 turned out to be primarily caused by file close issues --
specifically, the main I/O concentrator thread and its pool of worker threads
were not being shut down properly on file close. Addressing this issue in
combination with some other minor fixes seems to have addressed these issues.
Items 5 & 6 appear to have been caused by issue of I/O requests to the
thread pool in an order that did not maintain POSIX semantics. A rewrite of
the I/O request dispatch code appears to have solved these issues.
Item 7 seems to have been caused by multiple write requests from a given
rank being read by the wrong worker thread. Code to issue "unique" tags for
each write request via the ACK message appears to have cleaned this up.
Note that the code is still in poor condtition. A partial list of known
defects includes:
a) Race condiditon on file close that allows superblock writes to arrive
at the I/O concentrator after it has been shutdown. This defect is
most evident when testpar/t_subfiling_vfd is run with 8 ranks.
b) No error reporting from I/O concentrators -- must design and implement
this. For now, mostly just asserts, which suggests that it should be
run in debug mode.
c) Much commented out and/or un-used code.
d) Code orgnaization
e) Build system with bits of Mercury is awkward -- think of shifting
to pthreads with our own thread pool code.
f) Need to add native support for vector and selection I/O to the subfiling
VFD.
g) Need to review, and posibly rework configuration code.
h) Need to store subfile configuration data in a superblock extension message,
and add code to use this data on file open.
i) Test code is inadequate -- expect more issues as it is extended.
In particular, there is no unit test code for the I/O request dispatch code.
While I think it is correct at present, we need test code to verify this.
Similarly, we need to test with multiple I/O concentrators and much smaller
stripe depth.
My actual code changes were limited to:
src/H5FDioc.c
src/H5FDioc_threads.c
src/H5FDsubfile_int.c
src/H5FDsubfile_mpi.c
src/H5FDsubfiling.c
src/H5FDsubfiling.h
src/H5FDsubfiling_priv.h
testpar/t_subfiling_vfd.c
testpar/t_vfd.c
I'm not sure what is going on with the deletions in src/mercury/src/util.
Tested parallel/debug on Charis and Jelly
* subfiling with selection IO (#1219)
Merged branch 'selection_io' into subfiling branch.
* Subfile name fixes (#1250)
* fixed subfiling naming convention, and added leading zero to rank names.
* Merge branch 'selection_io' into selection_io_with_subfiling_vfd (#1265)
* Added script to join subfiles into a single HDF5 file (#1350)
* Modified H5FD__subfiling_query() to report that the sub-filing VFD supports MPI
This exposed issues with truncate and get EOF in the sub-filing VFD.
I believe I have addressed these issues (get EOF not as fully tested as it should be), howeer,
it exposed race conditions resulting in hangs. As of this writing, I have not been able
to chase these down.
Note that the tests that expose these race conditions are in testpar/t_subfiling_vfd.c, and
are currently skipped. Unskip these tests to reproduce the race conditions.
tested (to the extent possible) debug/parallel on charis and jelly.
* Committing clang-format changes
* fixed H5MM_free
Co-authored-by: mainzer <mainzer#hdfgroup.org>
Co-authored-by: jrmainzer <72230804+jrmainzer@users.noreply.github.com>
Co-authored-by: Richard Warren <Richard.Warren@hdfgroup.org>
Co-authored-by: Richard.Warren <richard.warren@jelly.ad.hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Move Subfiling VFD components into H5FDsubfiling source directory
* Update Autotools build and add H5_HAVE_SUBFILING_VFD macro to H5pubconf.h
* Tidy up CMake build of subfiling sources
* Merge branch 'develop' into feature/subfiling (#1539)
Merge branch 'develop' into feature/subfiling
* Add VFD interface version field to Subfiling and IOC VFDs
* Merge branch 'develop' into feature/subfiling (#1557)
Merge branch 'develop' into feature/subfiling
* Merge branch 'develop' into feature/subfiling (#1563)
Merge branch 'develop' into feature/subfiling
* Tidy up merge artifacts after rebase on develop
* Fix incorrect variable in mirror VFD utils CMake
* Ensure VFD values are always defined
* Add subfiling to CMake VFD_LIST if built
* Mark MPI I/O driver self-initialization global as static
* Add Subfiling VFD to predefined VFDs for HDF5_DRIVER env. variable
* Initial progress towards separating private vs. public subfiling code
* include libgen.h in t_vfd tests for correct dirname/basename
* Committing clang-format changes
* removed mercury option, included subfiling header path (#1577)
Added subfiling status to configure output, installed h5fuse.sh to build directory for use in future tests.
* added check for stdatomic.h (#1578)
* added check for stdatomic.h with subfiling
* added H5_HAVE_SUBFILING_VFD for cmake
* fix old-style-definition warning (#1582)
* fix old-style-definition warning
* added test for enable parallel with subfiling VFD (#1586)
Fails if subfiling VFD is not used with parallel support.
* Subfiling/IOC VFD fixes and tidying (#1619)
* Rename CMake option for Subfiling VFD to be consistent with other VFDs
* Miscellaneous Subfiling fixes
Add error message for unset MPI communicator
Support dynamic loading of subfiling VFD with default configuration
* Temporary fix for subfile name issue
* Added subfile checks (#1634)
* added subfile checks
* Feature/subfiling (#1655)
* Subfiling/IOC VFD cleanup
Fix misuse of MPI_COMM_WORLD in IOC VFD
Propagate Subfiling FAPL MPI settings down to IOC FAPL in default
configuration case
Cleanup IOC VFD debugging code
Change sprintf to snprintf in a few places
* Major work on separating Subfiling and IOC VFDs from each other
* Re-write async_completion func to not overuse stack
* Replace usage of MPI_COMM_WORLD with file's actual MPI communicator
* Refactor H5FDsubfile_mpi.c
* Remove empty file H5FDsubfile_mpi.c
* Separate IOC VFD errors to its own error stack
* Committing clang-format changes
* Remove H5TRACE macros from H5FDioc.c
* Integrate H5FDioc_threads.c with IOC error stack
* Fix for subfile name generation
Use number of I/O concentrators from existing subfiling configuration file, if one exists
* Add temporary barrier in "Get EOF" operation to prevent races on EOF
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Fix for retrieval of machine Host ID
* Default to MPI_COMM_WORLD if no MPI params set
* added libs rt and pthreads (#1673)
* added libs rt and pthreads
* Feature/subfiling (#1689)
* More tidying of IOC VFD and subfiling debug code
* Remove old unused log file code
* Clear FID from active file map on failure
* Fix bug in generation of subfile names when truncating file
* Change subfile names to start from 1 instead of 0
* Use long long for user-specified stripe size from environment variable
* Skip 0-sized I/Os in low-level IOC I/O routines
* Don't update EOF on read
* Convert printed warning about data size mismatch to assertion
* Don't add base file address to I/O addresses twice
Base address should already be applied as part of H5FDwrite/read_vector calls
* Account for 0-sized I/O vector entries in subfile write/read functions
* Rewrite init_indep_io for clarity
* Correction for IOC wraparound calculations
* Some corrections to iovec calculations
* Remove temporary barrier on EOF retrieval
* Complete work request queue entry on error instead of skipping over
* Account for stripe size wraparound for sf_col_offset calculation
* Committing clang-format changes
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Re-write and fix bugs in I/O vector filling routines (#1703)
* Rewrite I/O vector filling routines for clarity
* Fix bug with iovec_fill_last when last I/O size is 0
* added subfiling_dir line read (#1714)
* added subfiling_dir line read and use it
* shellcheck fixes
* I/O request dispatch logic update (#1731)
Short-circuit I/O request dispatch when head of I/O queue is an
in-progress get EOF or truncate operation. This prevents an issue where
a write operation can be dispatched alongside a get EOF/truncate
operation, whereas all I/O requests are supposed to be ineligible for
dispatch until the get EOF/truncate is completed
* h5fuse.sh.in clean-up (#1757)
* Added command-line options
* Committing clang-format changes
* Align with changes from develop
* Mimic MPI I/O VFD for EOF handling
* Initialize context_id field for work request objects
* Use logfile for some debugging information
* Use atomic store to set IOC ready flag
* Use separate communicator for sending file EOF data
Minor IOC cleanup
* Use H5_subfile_fid_to_context to get context ID for file in Subfiling
VFD
* IOVEC calculation fixes
* Updates for debugging code
* Minor fixes for threaded code
* Committing clang-format changes
* Use separate MPI communicator for barrier operations
* Committing clang-format changes
* Rewrite EOF routine to use nonblocking MPI communication
* Committing clang-format changes
* Always dispatch I/O work requests in IOC main loop
* Return distinct MPI communicator to library when requested
* Minor warning cleanup
* Committing clang-format changes
* Generate h5fuse.sh from h5fuse.sh.in in CMake
* Send truncate messages to correct IOC rank
* Committing clang-format changes
* Miscellaneous cleanup
Post some MPI receives before sends
Free some duplicated MPI communicator/Info objects
Remove unnecessary extra MPI_Barrier
* Warning cleanup
* Fix for leaked MPI communicator
* Retrieve file EOF on single rank and bcast it
* Fixes for a few failure paths
* Cleanup of IOC file opens
* Committing clang-format changes
* Use plan MPI_Send for send of EOF messages
* Always check MPI thread support level during Subfiling init
* Committing clang-format changes
* Handle a hang on failure when IOCs can't open subfiles
* Committing clang-format changes
* Refactor file open status consensus check
* Committing clang-format changes
* Fix for MPI_Comm_free being called after MPI_Finalize
* Fix VFD test by setting MPI params before setting subfiling on FAPL
* Update Subfiling VFD error handling and error stack usage
* Improvements for Subfiling logfiles
* Remove prototypes for currently unused routines
* Disable I/O queue stat collecting by default
* Remove unused serialization mutex variable
* Update VFD testing to take subfiling VFD into account
* Fix usage of global subfiling application layout object
* Minor fixes for failure pathways
* Keep track of the number of failures in an IOC I/O queue
* Make sure not to exceed MPI_TAG_UB value for data communication messages
* Committing clang-format changes
* Update for rename of some H5FD 'ctl' opcodes
* Always include Subfiling's public header files in hdf5.h
* Remove old unused code and comments
* Implement support for per-file I/O queues
Allows the subfiling VFD to have multiple HDF5 files open simultaneously
* Use simple MPI_Iprobe over unnecessary MPI_Improbe
* Committing clang-format changes
* Update HDF5 testing to query driver for H5FD_FEAT_DEFAULT_VFD_COMPATIBLE
flag
* Fix a few bugs related to file multi-opens
* Avoid calling MPI routines if subfiling gets reinitialized
* Fix issue when files are closed in a random order
* Update HDF5 testing to query VFD for "using MPI" feature flag
* Register atexit handler in subfiling VFD to call MPI_Finalize after HDF5
closes
* Fail for collective I/O requests until support is implemented
* Correct VOL test function prototypes
* Minor cleanup of old code and comments
* Update mercury dependency
* Cleanup of subfiling configuration structure
* Committing clang-format changes
* Build system updates for Subfiling VFD
* Fix possible hang on failure in t_vfd tests caused by mismatched
MPI_Barrier calls
* Copy subfiling IOC fapl in "fapl get" method
* Mirror subfiling superblock writes to stub file for legacy POSIX-y HDF5
applications
* Allow collective I/O for MPI_BYTE types and rank 0 bcast strategy
* Committing clang-format changes
* Use different scheme for subfiling write message MPI tag calculations
* Committing clang-format changes
* Avoid performing fstat calls on all MPI ranks
* Add MPI_Barrier before finalizing IOC threads
* Use try_lock in I/O queue dispatch to minimize contention from worker threads
* Use simple Waitall for nonblocking I/O waits
* Add configurable IOC main thread delay and try_lock option to I/O queue dispatch
* Fix bug that could cause serialization of non-overlapping I/O requests
* Temporarily treat collective subfiling vector I/O calls as independent
* Removed unused mercury bits
* Add stubs for subfiling and IOC file delete callback
* Update VFD testing for Subfiling VFD
* Work around HDF5 metadata cache bug for Subfiling VFD when MPI Comm size
= 1
* Committing clang-format changes
Co-authored-by: mainzer <mainzer#hdfgroup.org>
Co-authored-by: Neil Fortner <nfortne2@hdfgroup.org>
Co-authored-by: Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: jrmainzer <72230804+jrmainzer@users.noreply.github.com>
Co-authored-by: Richard Warren <Richard.Warren@hdfgroup.org>
Co-authored-by: Richard.Warren <richard.warren@jelly.ad.hdfgroup.org>
|