mirror of
https://github.com/HDFGroup/hdf5.git
synced 2026-09-25 04:09:44 +03:00
0743df227f
Adds workflow to build ROS3 VFD and optionally build aws-c-s3 library from source or use package managers Adds testing of ROS3 VFD with s3proxy and docker Adds new H5Pset_fapl_ros3_endpoint()/H5Pget_fapl_ros3_endpoint() API functions to set/get an alternative endpoint URL to use when opening files with the ROS3 VFD Cleans up warnings in tools and tests related to ROS3 VFD structure size Co-authored-by: Larry Knox <lrknox@hdfgroup.org> Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
186 lines
8.6 KiB
Plaintext
186 lines
8.6 KiB
Plaintext
*************************************************************************
|
|
* Build and testing instructions for HDF5's ROS3 VFD *
|
|
*************************************************************************
|
|
|
|
Table of Contents
|
|
|
|
Section I: Preconditions
|
|
Section II: Obtaining the aws-c-s3 library
|
|
Section III: Building the aws-c-s3 library from source
|
|
Section IV: Building HDF5 with the ROS3 VFD enabled
|
|
Section V: Testing the ROS3 VFD
|
|
|
|
************************************************************************
|
|
|
|
========================================================================
|
|
I. Preconditions
|
|
========================================================================
|
|
|
|
- Refer to release_docs/INSTALL for preconditions and instructions for
|
|
building HDF5
|
|
|
|
========================================================================
|
|
II. Obtaining the aws-c-s3 library
|
|
========================================================================
|
|
|
|
As of the HDF5 2.0.0 release (or HDF5 commit 75fff3ed0663d3617e3c5cc3c57c1f0c83b38be4),
|
|
HDF5's ROS3 VFD requires the aws-c-s3 library (https://github.com/awslabs/aws-c-s3)
|
|
in order to be built. This library is distributed by several package
|
|
managers, usually as 'aws-c-s3', but consult your particular package
|
|
manager for the name of this package if it's available. HDF5 installs
|
|
this library in the following ways in CI testing:
|
|
|
|
Linux (Ubuntu):
|
|
|
|
- from source, as Ubuntu doesn't have a package for aws-c-s3 yet
|
|
|
|
Windows:
|
|
|
|
- vcpkg install aws-c-s3 (make sure to set CMAKE_TOOLCHAIN_FILE to
|
|
point to vcpkg.cmake when configuring HDF5)
|
|
|
|
MacOS:
|
|
|
|
- brew install aws-c-s3
|
|
|
|
========================================================================
|
|
III. Building the aws-c-s3 library from source
|
|
========================================================================
|
|
|
|
If you are unable to obtain the aws-c-s3 library from a package manager
|
|
or other distribution method, the library may also be built from source
|
|
code. The following instructions have been reproduced and slightly
|
|
modified from the aws-c-s3 repository and may be out of date; be sure to
|
|
check https://github.com/awslabs/aws-c-s3 for the latest build
|
|
instructions.
|
|
|
|
1. Obtain the source code for aws-c-s3 and its dependencies
|
|
|
|
Note that this build process intends for aws-c-s3 and all its
|
|
dependencies to be cloned into the same top-level directory.
|
|
|
|
Only if building on Linux:
|
|
- git clone https://github.com/aws/aws-lc.git
|
|
- git clone https://github.com/aws/s2n-tls.git
|
|
|
|
- git clone https://github.com/awslabs/aws-c-common.git
|
|
- git clone https://github.com/awslabs/aws-checksums.git
|
|
- git clone https://github.com/awslabs/aws-c-cal.git
|
|
- git clone https://github.com/awslabs/aws-c-io.git
|
|
- git clone https://github.com/awslabs/aws-c-compression.git
|
|
- git clone https://github.com/awslabs/aws-c-http.git
|
|
- git clone https://github.com/awslabs/aws-c-sdkutils.git
|
|
- git clone https://github.com/awslabs/aws-c-auth.git
|
|
- git clone https://github.com/awslabs/aws-c-s3.git
|
|
|
|
2. Build aws-c-s3 and its dependencies
|
|
|
|
Note that this build process intends for <install-path> to be the
|
|
same directory for all commands. '--parallel <num-jobs>' is used
|
|
to speed up the build process by building with <num-jobs> concurrent
|
|
processes, and can be omitted if desired.
|
|
|
|
Only if building on Linux:
|
|
Build aws-lc:
|
|
- cmake -S aws-lc -B aws-lc/build -DCMAKE_INSTALL_PREFIX=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-lc/build --target install --parallel <num-jobs>
|
|
|
|
Build s2n-tls:
|
|
- cmake -S s2n-tls -B s2n-tls/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build s2n-tls/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-common:
|
|
- cmake -S aws-c-common -B aws-c-common/build -DCMAKE_INSTALL_PREFIX=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-common/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-checksums:
|
|
- cmake -S aws-checksums -B aws-checksums/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-checksums/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-cal:
|
|
- cmake -S aws-c-cal -B aws-c-cal/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-cal/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-io:
|
|
- cmake -S aws-c-io -B aws-c-io/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-io/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-compression:
|
|
- cmake -S aws-c-compression -B aws-c-compression/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-compression/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-http:
|
|
- cmake -S aws-c-http -B aws-c-http/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-http/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-sdkutils:
|
|
- cmake -S aws-c-sdkutils -B aws-c-sdkutils/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-sdkutils/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-auth:
|
|
- cmake -S aws-c-auth -B aws-c-auth/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-auth/build --target install --parallel <num-jobs>
|
|
|
|
Build aws-c-s3:
|
|
- cmake -S aws-c-s3 -B aws-c-s3/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
|
|
- cmake --build aws-c-s3/build --target install --parallel <num-jobs>
|
|
|
|
Note:
|
|
As shown in the above commands, it is suggested to pass the CMake
|
|
option BUILD_SHARED_LIBS=1 when building aws-c-s3 and each of its
|
|
dependencies if shared libraries will be enabled when building HDF5.
|
|
Otherwise, HDF5's CMake configuration logic might find and use a
|
|
libcrypto library from the system, which could result in linking errors
|
|
depending on the version of that library. If only static libraries will
|
|
be enabled when building HDF5, this option can be omitted.
|
|
|
|
Note:
|
|
The instructions above install aws-c-s3 and all of its dependencies
|
|
into the same directory with the use of CMAKE_INSTALL_PREFIX. This is
|
|
the suggested way of building aws-c-s3 as it simplifies the process of
|
|
locating aws-c-s3 and its dependencies with CMAKE_PREFIX_PATH, both
|
|
when building aws-c-s3 and HDF5, especially if installing aws-c-s3 to
|
|
a non-standard location.
|
|
|
|
========================================================================
|
|
IV. Building HDF5 with the ROS3 VFD enabled
|
|
========================================================================
|
|
|
|
HDF5 uses CMake's find_package() mechanism to locate the aws-c-library,
|
|
so as long as it is installed to a standard location, the CMake option
|
|
HDF5_ENABLE_ROS3_VFD=ON simply needs to be passed when configuring HDF5
|
|
in order to enable the ROS3 VFD, as in:
|
|
|
|
cmake -DHDF5_ENABLE_ROS3_VFD=ON ..
|
|
|
|
If the aws-c-s3 library is installed to a non-standard location, the
|
|
environment variable CMAKE_PREFIX_PATH should be set to that path when
|
|
configuring HDF5, as in:
|
|
|
|
CMAKE_PREFIX_PATH=<install-path> cmake -DHDF5_ENABLE_ROS3_VFD=ON ..
|
|
|
|
Refer to release_docs/INSTALL_CMake.txt for more general instructions on
|
|
building HDF5 with CMake.
|
|
|
|
========================================================================
|
|
V. Testing the ROS3 VFD
|
|
========================================================================
|
|
|
|
The ROS3 VFD is tested using a combination of docker, s3proxy and the
|
|
AWS CLI. If the 'docker' and 'aws' executables are available in standard
|
|
locations which CMake can find and the docker daemon is running, the
|
|
ROS3 VFD tests can be enabled by passing the CMake option
|
|
HDF5_ENABLE_ROS3_VFD_DOCKER_PROXY=ON, as in:
|
|
|
|
cmake -DHDF5_ENABLE_ROS3_VFD=ON -DHDF5_ENABLE_ROS3_VFD_DOCKER_PROXY=ON ..
|
|
|
|
If everything configured correctly, the ROS3 VFD tests can be run with:
|
|
|
|
ctest -R "S3TEST" -VV .
|
|
|
|
These tests are separated into different groups based on the particular
|
|
functionality being tested. For each group of tests, the CMake logic
|
|
will bring up s3proxy, create a testing bucket and populate that bucket
|
|
with any testing files needed, run all of the tests for the testing
|
|
group and then bring down s3proxy.
|