* Re-write ROS3 VFD S3 backend to use AWS' C S3 library
Replaces the ROS3 VFD's libcurl-based S3 backend with a new
backend which uses the aws-c-s3 library
Adds support for sourcing credentials from standard AWS
configuration files, environment variables, etc. with a
fallback to anonymous credentials when credentials can't be
found
Adds support for sourcing AWS region from standard AWS
configuration files and environment variables
Adds support for new environment variables HDF5_ROS3_VFD_DEBUG,
HDF5_ROS3_VFD_LOG_LEVEL and HDF5_ROS3_VFD_LOG_FILE for debugging
purposes
Replaces function to load testing credentials in the ros3.c and
s3comms.c tests with new function that delegates to the aws-c-s3
library for better compatibility with standard AWS configuration
files and environment variables
Adds support for s3:// object URIs
Adds HTTP User-Agent header in form of
'libhdf5/<hdf5_version> (vfd:ros3) libaws-c-s3'
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
* Refactor warning macros
Replaces the scheme in H5warnings.h with a more compiler-agnostic
one that relocates ifdef complexity to the warnings header file
and uses better naming.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Adds new -testexpress command-line option to the testframe testing
framework to allow setting or overriding of the TestExpress level
at runtime
Adds macros for the different currently defined TestExpress levels
Added infrastructure support for multithreaded concurrency by adding an optional way to switch to using a non-recursive R/W lock for the global API lock. This is enabled with a new 'concurrency' configuration flag for the autotools & CMake builds, which is disabled by default.
When the 'concurrency' build option is chosen, the global API lock will use the R/W lock and all API calls currently will acquire a write lock, ensuring exclusive access by one thread. Over time, the API routines that are converted to support multithreaded concurrency will switch to acquiring a read lock instead.
Reentering the library from application callbacks is managed by the 'disable locking for this thread' (DLFTT) threadsafety protocol. This is internally handled within the H5_API_LOCK / H5_API_UNLOCK macros in H5private.h (as before), which invoke the 'dlftt' routines in H5TSint.c.
To support this change, the threadsafety configuration macros for the library have been updated:
- --enable-threadsafe now defines the H5_HAVE_THREADSAFE macro
- --enable-concurrency defines the H5_HAVE_CONCURRENCY macro
The new H5_HAVE_THREADSAFE_API macro is set if either H5_HAVE_THREADSAFE or H5_HAVE_CONCURRENCY is enabled.
New Github actions are added to include the concurrency configuration in the CI for the develop branch.
To support the new non-recursive R/W locking for API routines, some other changes are necessary:
Added macro wrappers around all callback invocations that could call an
application function, and therefore re-enter the library:
H5_BEFORE_USER_CB* / H5_AFTER_USER_CB*
Added H5_user_cb_prepare / H5_user_cb_restore routines that save the
state of the library when callback leaves the library. Includes error
stack and threadsafe reentry state currently.
There's also some small cleanups to various places in the library:
Moved the H5E_mpi_error_str / H5E_mpi_error_str_len globals to be local for
pushing MPI errors, so that multiple threads can't interfere with each
other.
Added H5TS_rwlock_trywrlock() routine to R/W lock interface.
Emulate R/W locks on MacOS because its implementation of
pthread_rwlock_wrlock() does not conform to the POSIX standard.
Don't acquire the global API lock in H5close, since it's acquired in H5_term_library, which is necessary because H5_term_library is invoked via other code paths that don't hold the global API lock.
Don't call H5Eget_auto2 API routine within H5_term_library.
Switched 'return NULL' in H5allocate_memory to HGOTO_DONE(NULL).
Switched H5Pget_file_space_strategy / H5Pset_file_space_strategy to use
internal routines instead of API routines.
Switched H5Oopen_by_addr & H5Ovisit1 to use internal routines instead of
API routines.
Fixed a few places in src/H5Odeprec.c where a major
error ID was passed as a minor ID.
* Cast away a signed/unsigned issue in H5HFhuge.c
* Use our platform-independent POSIX types in external_common.c
and h5test.c
* Replace a memset() call with a (const) array initializer in h5test.c
* Fix an unused done: target in h5diff
This is where most people will expect to find license information. The
COPYING_LBNL_HDF5 file has also been renamed to LICENSE_LBNL_HDF5.
The licenses are unchanged.
Added new testframe.h header to document testing framework functions and
split them away from h5test.h and from test programs that don't
integrate with the testframe.c testing framework
Added new test setup callback to testframe.c testing framework
Added parameters to AddTest() to specify size of test parameters so they
can be copied for later use
Enabled HDF5 error stacks in testframe.c framework by default and added
some error stack suppressions to some testhdf5 tests
Added new maxthreads option to testframe.c framework to allow specifying
the maximum number of threads a multi-threaded test can use
Moved TestExpress functionality out of testframe.c and into more general
h5test.c for wider use by tests through getter and setter
Updated some tests to not mix and match functionality between h5test.c/h
and testframe.c/h
Moved some functionality from testphdf5.h into testpar.h for parallel
tests that aren't part of testphdf5
Added new parallel test library that contains common shared
functionality for parallel tests (similar to h5test library)
The t_pmulti_dset and t_select_io_dset tests rely on the behavior
of the previous private rand_r-like implementation to get the
correct sequence of random numbers to pass. This has been restored
using a fully private rand_r-like implementation that doesn't
rely on rand_r and will work on Windows and other platforms
where rand_r doesn't exist.
Complete overhaul of the concurrency-related aspects of the library (threading, atomics, locking, etc.), adding private routines in the H5TS package to allow internal algorithms to use all of these capabilities.
Adds many new features & components in the H5TS package that are equivalent to common concurrency data structures and capabilities: "regular" and recursive mutices, condition variables, semaphores, thread barriers, 'once' support, thread pools, atomic variables, thread-local keys, and spawning & joining internal threads.
Now supports C11, pthreads, and Windows threading for all H5TS capabilities, except the recursive readers/writers lock, which is not supported on Windows (because Windows threads don't provide a callback on thread-local variable deletion).
The "global" API lock is switched to use a recursive mutex from the H5TS package, instead of its own variant.
API context code (H5CX package) and error stacks (H5E package) now use the common thread-local info, instead of their own variants.
Subfiling code is switched from using Mercury threading features to the new internal H5TS features.
Removes the mercury threading code.
Adds a configure option (--enable-threads / HDF5_ENABLE_THREADS), enabled by default, to control whether threading is enabled within the library.
Changes Autotools testing to use HDF5_TEST_DRIVER environment
variable to avoid running tests that don't work well with several
VFDs
Restores old h5_get_vfd_fapl() testing function to setup a FAPL
with a particular VFD
Adds a macro for the default VFD name
The buffers passed to stat-like calls are only partially filled in by
the call, leaving ununitialized memory areas when the stat buffers are
created on the stack.
This change memsets the buffers to 0 before the stat calls, quieting
the -fsanitze=memory complaints.
Fixes an issue where the HDF5_USE_FILE_LOCKING environment variable being
set can interfere with the file locking setting that the test expects to
be returned.
off_t is a 32-bit signed value on Windows, so we should use HDoff_t
(which is __int64 on Windows) internally instead.
Also defines HDftell on Windows to be _ftelli64().
* HDF5 API test updates
Removed test duplication from bringing API tests
back into the library from external VOL tests
repo
Synced changes between API tests and library's
tests
Updated API tests CMake code to directly use and
install testhdf5, testphdf5, etc. instead of
creating duplicate binaries
Added new h5_using_native_vol() test function to
determine whether the VOL connector being used
is (or the VOL connector stack being used resolves
to) the native VOL connector
* Remove duplicate variable
* Strip HD prefix from string/char C API calls
* HD(f)(put|get)(s|c)
* HDstr*
* HDv*printf
* HD(s)(print|scan)f
* HDperror
But NOT:
* HDstrcase*
* HDvasprintf
* HDstrtok_r
* HDstrndup
As those are not C99 and have portability work-around
implementations. They will be handled later.
* Fix th5_system.c screwup
These calls are non-C99 but protected by ifdefs and have no Windows
equivalents:
* HDalarm
* HDasprintf
* HDclock_gettime
* HDfcntl
* HDgethostname
* HDgetrusage
* HDsymlink
* Adds semicolons to function-like macros
* Adds a do..while(0) loop to some macros
* Removes semicolons when inappropriate, especially H5E_TRY_BEGIN/END
* H5F_LIBVER_LATEST changes for move to 1.15
* Add new default api check
* Format fixes
* Fix default configure
* fix lib version tests
* Fix another version variable
* Add 1.14 doc link
* Updated source file copyright headers to remove "Copyright by the Board of Trustees
of the University of Illinois", which is kept in the top-level COPYING file.
Dana Robinsonandgithub-actions
Quincey Koziol
Allen Byrne
H. Joe Lee
Scot Breitenfeld
Sean McBride