mirror of https://github.com/HDFGroup/hdf5.git synced 2026-09-25 04:09:44 +03:00

Files

T

Scot Breitenfeld 05676d1abe Consolidate documentation under docs/ directory (#6310 )

* Consolidate documentation under doc/ directory

Move user-facing guides from release_docs/ and doxygen/ into a single
doc/ root. release_docs/ now holds only release artifacts (changelogs,
history, release process, maintainer info).

- git mv release_docs/INSTALL*.md, USING_*.md, README_HPC.md,
  BuildSystemNotes.md, AutotoolsToCMakeOptions.md,
  HDF5_Library_2.0.0_Migration_Guide.md → doc/
- git mv doxygen/ → doc/doxygen/
- Update CMakeLists.txt: HDF5_DOXYGEN_DIR and add_subdirectory path
- Update CMakeInstallation.cmake: all install paths for moved files
- Update bin/make_vers: hardcoded doxygen/ path substitution
- Update doc/doxygen/CMakeLists.txt: EXAMPLES_DIRECTORY and comments
- Update README.md, CONTRIBUTING.md, SECURITY.md, config/README.md,
  release_docs/RELEASE_PROCESS.md: links to moved files
- Update doxygen .dox files: release_docs/ URLs for moved guides
- Rewrite release_docs/README.md for narrowed scope

* Add HDF5_DOCS_DIR variable for doc/ root path

Introduce HDF5_DOCS_DIR = \${HDF5_SOURCE_DIR}/doc so that
CMakeInstallation.cmake and future callers reference the doc/
directory symbolically rather than by hardcoded path.
HDF5_DOXYGEN_DIR is now derived from HDF5_DOCS_DIR.

2026-03-31 17:01:54 -06:00

8.1 KiB

Raw Permalink Blame History

Build and Testing Instructions for HDF5's ROS3 VFD

NOTE: HDF5 2.0 and later requires CMake for building. See INSTALL_CMake.md for complete build instructions using CMake Presets.

Section I: Preconditions
Section II: Obtaining the aws-c-s3 library
Section III: Building the aws-c-s3 library from source
Section IV: Building HDF5 with the ROS3 VFD enabled
Section V: Testing the ROS3 VFD

I. Preconditions

Refer to INSTALL.md for preconditions and instructions for building HDF5.

II. Obtaining the `aws-c-s3` library

As of the HDF5 2.0.0 release (or HDF5 commit 75fff3ed0663d3617e3c5cc3c57c1f0c83b38be4), HDF5's ROS3 VFD requires the aws-c-s3 library in order to be built. This library is distributed by several package managers, usually as aws-c-s3, but consult your particular package manager for the name of this package if it's available.

HDF5 installs this library in the following ways in CI testing.

Linux (Ubuntu): From source, as Ubuntu doesn't have a package for aws-c-s3 yet.
Windows:
```
vcpkg install aws-c-s3
```
(Make sure to set CMAKE_TOOLCHAIN_FILE to point to vcpkg.cmake when configuring HDF5)
MacOS:
```
brew install aws-c-s3
```

III. Building the aws-c-s3 library from source

If you are unable to obtain the aws-c-s3 library from a package manager or other distribution method, the library may also be built from source code. The following instructions have been reproduced and slightly modified from the aws-c-s3 repository and may be out of date; be sure to check https://github.com/awslabs/aws-c-s3 for the latest build instructions.

1. Obtain the source code for aws-c-s3 and its dependencies

Note: This build process intends for aws-c-s3 and all its dependencies to be cloned into the same top-level directory.

# Only if building on Linux:
git clone https://github.com/aws/aws-lc.git
git clone https://github.com/aws/s2n-tls.git

# All platforms:
git clone https://github.com/awslabs/aws-c-common.git
git clone https://github.com/awslabs/aws-checksums.git
git clone https://github.com/awslabs/aws-c-cal.git
git clone https://github.com/awslabs/aws-c-io.git
git clone https://github.com/awslabs/aws-c-compression.git
git clone https://github.com/awslabs/aws-c-http.git
git clone https://github.com/awslabs/aws-c-sdkutils.git
git clone https://github.com/awslabs/aws-c-auth.git
git clone https://github.com/awslabs/aws-c-s3.git

2. Build `aws-c-s3` and its dependencies

Note: This build process intends for <install-path> to be the same directory for all commands. --parallel <num-jobs> is used to speed up the build process by building with <num-jobs> concurrent processes, and can be omitted if desired.

# ==========================================
# Only if building on Linux:
# ==========================================

# Build aws-lc:
cmake -S aws-lc -B aws-lc/build -DCMAKE_INSTALL_PREFIX=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-lc/build --target install --parallel <num-jobs>

# Build s2n-tls:
cmake -S s2n-tls -B s2n-tls/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build s2n-tls/build --target install --parallel <num-jobs>

# ==========================================
# All platforms:
# ==========================================

# Build aws-c-common:
cmake -S aws-c-common -B aws-c-common/build -DCMAKE_INSTALL_PREFIX=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-common/build --target install --parallel <num-jobs>

# Build aws-checksums:
cmake -S aws-checksums -B aws-checksums/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-checksums/build --target install --parallel <num-jobs>

# Build aws-c-cal:
cmake -S aws-c-cal -B aws-c-cal/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-cal/build --target install --parallel <num-jobs>

# Build aws-c-io:
cmake -S aws-c-io -B aws-c-io/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-io/build --target install --parallel <num-jobs>

# Build aws-c-compression:
cmake -S aws-c-compression -B aws-c-compression/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-compression/build --target install --parallel <num-jobs>

# Build aws-c-http:
cmake -S aws-c-http -B aws-c-http/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-http/build --target install --parallel <num-jobs>

# Build aws-c-sdkutils:
cmake -S aws-c-sdkutils -B aws-c-sdkutils/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-sdkutils/build --target install --parallel <num-jobs>

# Build aws-c-auth:
cmake -S aws-c-auth -B aws-c-auth/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-auth/build --target install --parallel <num-jobs>

# Build aws-c-s3:
cmake -S aws-c-s3 -B aws-c-s3/build -DCMAKE_INSTALL_PREFIX=<install-path> -DCMAKE_PREFIX_PATH=<install-path> -DBUILD_SHARED_LIBS=1
cmake --build aws-c-s3/build --target install --parallel <num-jobs>

Note: As shown in the above commands, it is suggested to pass the CMake option BUILD_SHARED_LIBS=1 when building aws-c-s3 and each of its dependencies if shared libraries will be enabled when building HDF5. Otherwise, HDF5's CMake configuration logic might find and use a libcrypto library from the system, which could result in linking errors depending on the version of that library. If only static libraries will be enabled when building HDF5, this option can be omitted.

Note: The instructions above install aws-c-s3 and all of its dependencies into the same directory with the use of CMAKE_INSTALL_PREFIX. This is the suggested way of building aws-c-s3 as it simplifies the process of locating aws-c-s3 and its dependencies with CMAKE_PREFIX_PATH, both when building aws-c-s3 and HDF5, especially if installing aws-c-s3 to a non-standard location.

IV. Building HDF5 with the ROS3 VFD enabled

HDF5 uses CMake's find_package() mechanism to locate the aws-c-s3 library, so as long as it is installed to a standard location, the CMake option HDF5_ENABLE_ROS3_VFD=ON simply needs to be passed when configuring HDF5 in order to enable the ROS3 VFD, as in:

cmake -DHDF5_ENABLE_ROS3_VFD=ON ..

If the aws-c-s3 library is installed to a non-standard location, the environment variable CMAKE_PREFIX_PATH should be set to that path when configuring HDF5, as in:

CMAKE_PREFIX_PATH=<install-path> cmake -DHDF5_ENABLE_ROS3_VFD=ON ..

Refer to INSTALL_CMake.md for more general instructions on building HDF5 with CMake.

V. Testing the ROS3 VFD

The ROS3 VFD is tested using a combination of docker, s3proxy and the AWS CLI. If the docker and aws executables are available in standard locations which CMake can find and the docker daemon is running, the ROS3 VFD tests can be enabled by passing the CMake option HDF5_ENABLE_ROS3_VFD_DOCKER_PROXY=ON, as in:

cmake -DHDF5_ENABLE_ROS3_VFD=ON -DHDF5_ENABLE_ROS3_VFD_DOCKER_PROXY=ON ..

If everything configured correctly, the ROS3 VFD tests can be run with:

ctest -R "S3TEST" -VV .

These tests are separated into different groups based on the particular functionality being tested. For each group of tests, the CMake logic will bring up s3proxy, create a testing bucket and populate that bucket with any testing files needed, run all of the tests for the testing group and then bring down s3proxy.

8.1 KiB Raw Permalink Blame History