Adds workflow to build ROS3 VFD and optionally build aws-c-s3 library
from source or use package managers
Adds testing of ROS3 VFD with s3proxy and docker
Adds new H5Pset_fapl_ros3_endpoint()/H5Pget_fapl_ros3_endpoint() API
functions to set/get an alternative endpoint URL to use when opening
files with the ROS3 VFD
Cleans up warnings in tools and tests related to ROS3 VFD structure size
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
* Re-write ROS3 VFD S3 backend to use AWS' C S3 library
Replaces the ROS3 VFD's libcurl-based S3 backend with a new
backend which uses the aws-c-s3 library
Adds support for sourcing credentials from standard AWS
configuration files, environment variables, etc. with a
fallback to anonymous credentials when credentials can't be
found
Adds support for sourcing AWS region from standard AWS
configuration files and environment variables
Adds support for new environment variables HDF5_ROS3_VFD_DEBUG,
HDF5_ROS3_VFD_LOG_LEVEL and HDF5_ROS3_VFD_LOG_FILE for debugging
purposes
Replaces function to load testing credentials in the ros3.c and
s3comms.c tests with new function that delegates to the aws-c-s3
library for better compatibility with standard AWS configuration
files and environment variables
Adds support for s3:// object URIs
Adds HTTP User-Agent header in form of
'libhdf5/<hdf5_version> (vfd:ros3) libaws-c-s3'
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
Fix an incorrect range check that would prevent reads
from being served from the ROS3 VFD's internal cache
when the offset + length of the read extends exactly
to the end of the cached range of bytes.
Build test generator programs against shared libraries, if available,
by default. Otherwise, build them against static libraries. If
HDF5_BUILD_STATIC_TOOLS is set to ON, the tools test generator
programs will be built against static libraries even if shared
libraries are available.
Moves call to TestAlarmOn() in testframe framework back into
the PerformTests() function due to the timing of when the
framework processes command-line arguments
Adds new -testexpress command-line option to the testframe testing
framework to allow setting or overriding of the TestExpress level
at runtime
Adds macros for the different currently defined TestExpress levels
Makes testing parameters specified with AddTest non-const again when
passed to test functions. This allows test functions to update their
parameters during execution and is particularly useful for tests
which use those parameters to accumulate statistics, error counts,
etc. without resorting to global variables.
* Implement support for complex number datatypes
Adds the new datatype class H5T_COMPLEX
Adds the new API function H5Tcomplex_create which creates a complex
number datatype from an ID of a base floating-point datatype
Adds the new feature check macros H5_HAVE_COMPLEX_NUMBERS and
H5_HAVE_C99_COMPLEX_NUMBERS
Adds the new datatype size macros H5_SIZEOF_FLOAT_COMPLEX,
H5_SIZEOF_DOUBLE_COMPLEX and H5_SIZEOF_LONG_DOUBLE_COMPLEX
Adds the new datatype ID macros H5T_NATIVE_FLOAT_COMPLEX,
H5T_NATIVE_DOUBLE_COMPLEX, H5T_NATIVE_LDOUBLE_COMPLEX,
H5T_CPLX_IEEE_F16LE, H5T_CPLX_IEEE_F16BE,
H5T_CPLX_IEEE_F32LE, H5T_CPLX_IEEE_F32BE,
H5T_CPLX_IEEE_F64LE and H5T_CPLX_IEEE_F64BE
Adds hard and soft datatype conversion paths between complex number
datatypes and all the integer and floating-point datatypes, as well as
between other complex number datatypes
Adds a special conversion path between complex number datatypes and
array or compound datatypes where the in-memory layout of data is the
same between the datatypes and data can be converted directly
Adds support for complex number datatypes to the h5dump, h5ls and
h5diff/ph5diff tools. Allows h5dump '-m' option to change floating-point
printing format for float complex and double complex datatypes, as well
as long double complex if it has the same size as double complex
Adds minimal support to the h5watch and h5import tools
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the Java wrappers. Also adds initial,
untested support to the JNI for future use with HDFView
Adds support for just the H5T_COMPLEX datatype class to the Fortran
wrappers
Adds support for the predefined complex number datatypes and
H5Tcomplex_create function to the high level library H5LT interface
for use with the H5LTtext_to_dtype and H5LTdtype_to_text functions
Changes some usages of "complex" in the library since it conflicts with
the "complex" keyword from the complex.h header. Also changes various
usages of the word "complex" throughout the library to distinguish
compound datatypes from complex datatypes.
Added setup and cleanup callback parameters to TestInit() to perform
setup and cleanup tasks once for whole test program
Removed TestCleanup() function since its functionality is covered by
PerformTests()
Added check of the HDF5_NOCLEANUP environment variable in
GetTestCleanup()
Added new testframe.h header to document testing framework functions and
split them away from h5test.h and from test programs that don't
integrate with the testframe.c testing framework
Added new test setup callback to testframe.c testing framework
Added parameters to AddTest() to specify size of test parameters so they
can be copied for later use
Enabled HDF5 error stacks in testframe.c framework by default and added
some error stack suppressions to some testhdf5 tests
Added new maxthreads option to testframe.c framework to allow specifying
the maximum number of threads a multi-threaded test can use
Moved TestExpress functionality out of testframe.c and into more general
h5test.c for wider use by tests through getter and setter
Updated some tests to not mix and match functionality between h5test.c/h
and testframe.c/h
Moved some functionality from testphdf5.h into testpar.h for parallel
tests that aren't part of testphdf5
Added new parallel test library that contains common shared
functionality for parallel tests (similar to h5test library)
Fixes a bug where the vector I/O sizes weren't being extended when
one of the entries in the array is 0. This caused an over-read of
the I/O sizes buffer and on some machines would cause a memory
allocation failure due to the calculated I/O vector size being too
large.
* Fixes and cleanup for ph5diff
Fixes concurrency issues in ph5diff that can cause interleaved
output
Fixes an issue where output can sometimes be dropped if it ended
up in ph5diff's output overflow file
Fixes an issue where MPI_Init is called after HDF5 has been
initialized, preventing the library from setting up an MPI
attribute to perform cleanup on MPI_Finalize
Fixes an issue in config/cmake/runTest.cmake where the CMake
logic would try to access an invalid list index if the number
of lines in a test's output and reference files don't match
* Add release note
Run configure checks with and without CFLAGS/CMAKE_C_FLAGS since some
compilers work in one case while not working in the other case
Sync CMake configure checks with Autotools
Fixes an issue where invalid datatype IDs are passed to application conversion
functions in the case where the top-level conversion function is a library-internal
function that operates on a container-like datatype, but one or more of the
base datatype members are converted with an application conversion function.
Converts an assertion in H5G_loc_find into a normal error
check that checks for empty link names
Initializes H5O_link_t structure early in H5G__ent_to_link
to avoid trying to free potentially uninitialized memory
Checks for an empty link name after H5MM_strndup in
H5G__ent_to_link
Fixes GitHub #4307
Add configure option to enable or disable extension features in general
Add configure option to enable or disable _Float16 support
Add new config options to various settings files
H5PB_read previously did not account for the fact that the size of the
read it's performing could overflow the page buffer pointer, depending
on the calculated offset for the read. This has been fixed by adjusting
the size of the read if it's determined that it would overflow the page.
When the H5_addr_overlap macro was updated to use H5_RANGE_OVERLAP,
it failed to take into account that H5_RANGE_OVERLAP expects the
range to be inclusive. This lead to an assertion failure in
H5MM_memcpy due to a memcpy operation on overlapping memory.
This has been fixed by subtracting 1 from the calculated high
bound values passed to H5_RANGE_OVERLAP
Changes Autotools testing to use HDF5_TEST_DRIVER environment
variable to avoid running tests that don't work well with several
VFDs
Restores old h5_get_vfd_fapl() testing function to setup a FAPL
with a particular VFD
Adds a macro for the default VFD name
* Fix issue with Subfiling VFD and multiple opens of same file
* Update H5_subfile_fid_to_context to return error value instead of ID
* Add helper routine to initialize open file mapping
Removes some datatype copying calls that are now unnecessary after
refactoring the datatype conversion code to use pointers internally
rather than IDs
Rewrites the enum conversion function so that it uses cached copies
of the source and destination datatypes in order to avoid modifying
the datatypes passed in
Adds a 'recursive' field to the datatype conversion context which
allows the conversion functions for members of a container datatype
to skip unnecessary repetitive conversion setup code
Changes internal datatype conversion callback functions so that the
source and destination datatype structure pointers are const
Removes some unused and unnecessary internal IDs registered with
H5I_register
Fixed some conversion issues with Clang due to problematic undefined
behavior when casting a negative floating-point value to an integer
Fixed a bug in the library's software integer to floating-point
conversion function where a user's conversion exception function
returning H5T_CONV_UNHANDLED in the case of overflows would result in
incorrect data after conversion
Added configure checks for functions and macros related to _Float16
usage since some compilers expose the datatype but not the functions or
macros
Fixed a dt_arith test failure when H5_WANT_DCONV_EXCEPTION isn't defined
Fixed a few warnings from not explicitly casting some _Float16 variables
upwards
Makes the datatype conversion context object available during both the
initialization and conversion processes for a datatype conversion
function, allowing the compound, variable-length and array datatype
conversion functions to avoid creating IDs for the datatypes when they
aren't necessary
Adds internal H5CX_pushed routine to determine if an API context is
available to retrieve values from
Also adds error checking to several places in H5T.c and H5Tconv.c where
the code had previously assumed object close operations would succeed
The datatype conversion code previously used IDs for the source and
destination datatypes rather than pointers to the internal structures
for those datatypes. This was mostly due to the need for an ID for these
datatypes that can be passed to an application-registered datatype
conversion function or datatype conversion exception function. However,
using IDs internally caused a lot of unnecessary ID lookups and hurt
performance of datatype conversions in general. This was especially
problematic for compound datatype conversions, where the ID lookups were
occuring on every member of every compound element of a dataset. The
code has now been refactored to use pointers internally and only create
IDs for datatypes when necessary.
Fixed a test issue in dt_arith where a library datatype conversion
function was being cast to an application conversion function. Since the
two have different prototypes, this started failing after the parameters
for a library conversion function changed from hid_t to H5T_t * and an
extra parameter was added. This appears to have worked coincidentally in
the past since the only different between a library conversion function
and application conversion function was an extra DXPL parameter at the
end of an application conversion function
Fixed an issue where memory wasn't being freed in the h5fc_chk_idx test
program. Even though the program exits quickly after allocating the
memory, it still causes failures when testing with -fsanitize=address
When clearing out datatype conversion paths involving variable-length or reference datatypes
on file close, also check for these datatypes inside compound or array datatypes
jhendersonHDF
