mirrors/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2026-09-25 04:09:44 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
bc8f6bb02aaa8b162bf04b42f2b59bcbbbfcf9f8
hdf5/release_docs/BuildSystemNotes.md
T
Allen Byrne b754dcb8f2 Move Java wrappers to FFM using jextract and java 25 (#5957) 
FFM build requires Java 25, Jextract 25.
Generates FFM bindings during configure.
JNI is default when the requirements are not met or can be forced.
Presets added for maven and FFM - JNI is default selection.
Enhanced Maven options will work with either JNI or FFM
New Workflows for testing and maven uploads.
Extensive documentation changes for java.
2025-11-04 14:03:06 -06:00

261 lines
12 KiB
Markdown
Raw Blame History

# HDF5 Build System Summary
A comprehensive analysis of the HDF5 CMake-only build system and CI/CD infrastructure for development planning.
## Build System Architecture
### CMake-Only Migration (March 2025)
- **Complete transition**: Autotools support was completely dropped in HDF5 2.0
- **CMake minimum version**: 3.26 required
- **Out-of-source builds**: Enforced - in-source builds are blocked with clear error messages
### Core CMake Structure
#### Main Configuration Files
- `CMakeLists.txt` - Root build configuration
- `CMakeBuildOptions.cmake` - Centralized build option definitions
- `CMakeFilters.cmake` - Compression filter support (zlib, szip, libaec)
- `CMakeTests.cmake` - Testing infrastructure configuration
- `CMakeInstallation.cmake` - Installation and packaging setup
- `CMakeVOL.cmake` - Virtual Object Layer connector support
- `CMakePlugins.cmake` - Plugin architecture support
- `java/src/hdf/hdf5lib/pom.xml.in` - Maven POM template for Java artifacts
#### Build Options Categories
**Library Types:**
- `BUILD_STATIC_LIBS=ON` - Static library builds
- `BUILD_SHARED_LIBS=ON` - Shared library builds
- `HDF5_ONLY_SHARED_LIBS=OFF` - Force shared-only builds
- `HDF5_BUILD_STATIC_TOOLS=OFF` - Static vs shared tools
**Language Bindings:**
- `HDF5_BUILD_CPP_LIB=OFF` - C++ bindings
- `HDF5_BUILD_FORTRAN=OFF` - Fortran bindings
- `HDF5_BUILD_JAVA=OFF` - Java bindings
- `HDF5_ENABLE_MAVEN_DEPLOY=OFF` - Maven repository deployment support
- `HDF5_MAVEN_SNAPSHOT=OFF` - Build Maven snapshot versions with -SNAPSHOT suffix
**Core Features:**
- `HDF5_ENABLE_PARALLEL=OFF` - MPI parallel I/O support
- `HDF5_ENABLE_THREADSAFE=OFF` - Thread safety (mutually exclusive with parallel)
- `HDF5_ENABLE_CONCURRENCY=OFF` - Multi-threaded concurrency
- `HDF5_BUILD_HL_LIB=ON` - High-level APIs
- `HDF5_BUILD_TOOLS=ON` - Command-line utilities
**Compression & Filters:**
- `HDF5_ENABLE_ZLIB_SUPPORT=OFF` - DEFLATE compression
- `HDF5_ENABLE_SZIP_SUPPORT=OFF` - SZIP compression
- `HDF5_ENABLE_PLUGIN_SUPPORT=OFF` - Runtime plugin loading
- `HDF5_USE_ZLIB_NG=OFF` - Use zlib-ng instead of zlib
**Advanced Features:**
- `HDF5_ENABLE_SUBFILING_VFD=OFF` - Parallel subfiling VFD
- `HDF5_ENABLE_MAP_API=OFF` - Map API (experimental)
- `HDF5_ENABLE_HDFS=OFF` - Hadoop HDFS support
## CMake Preset System
### Preset Architecture
- **Layered inheritance**: Base presets + feature-specific + platform-specific
- **Hidden presets**: Reusable components (`ci-base`, `ci-Debug`, `ci-Release`, `ci-Maven`, `ci-Maven-Snapshot`, `ci-Maven-Minimal`, `ci-Maven-Minimal-Snapshot`)
- **Platform presets**: `ci-GNUC`, `ci-Clang`, `ci-MSVC`, `ci-macos`
- **Maven presets**: Hidden base configurations for Maven deployment support
- **Minimal Maven presets**: Streamlined configurations for Java artifact generation only
- **Build type matrix**: Debug, Release (RelWithDebInfo + docs), Maven variants
### Key Preset Patterns
```bash
# Standard shared library builds
cmake --workflow --preset ci-StdShar-GNUC --fresh # GCC
cmake --workflow --preset ci-StdShar-Clang --fresh # Clang
cmake --workflow --preset ci-StdShar-MSVC --fresh # MSVC
# Maven-enabled builds (Java artifacts with deployment support)
cmake --workflow --preset ci-StdShar-GNUC-Maven --fresh # Maven release (full build)
cmake --workflow --preset ci-StdShar-GNUC-Maven-Snapshot --fresh # Maven snapshot (full build)
cmake --workflow --preset ci-MinShar-GNUC-Maven --fresh # Maven release (minimal build)
cmake --workflow --preset ci-MinShar-GNUC-Maven-Snapshot --fresh # Maven snapshot (minimal build)
# Multi-platform Maven presets (minimal builds for Java artifacts only)
cmake --workflow --preset ci-MinShar-MSVC-Maven --fresh # Windows Maven
cmake --workflow --preset ci-MinShar-Clang-Maven --fresh # macOS Maven
# Naming convention: ci-[Features]-[Compiler][-Maven[-Snapshot]]
# Features: Std (standard), Min (minimal), StdShar (standard shared), MinShar (minimal shared)
# Maven: Adds Maven deployment support with platform-specific JARs
# Snapshot: Adds -SNAPSHOT suffix for development versions
# Minimal Maven presets: Skip examples, testing, tools, C++, Fortran - Java artifacts only
# Java Examples Maven Integration: Comprehensive testing of Java examples with Maven artifacts across all platforms
```
### Preset Configuration Strategy
- **Binary directory**: `${sourceParentDir}/build/${presetName}`
- **Install directory**: `${sourceParentDir}/install/${presetName}`
- **Generator**: Ninja (default for most presets)
- **External libraries**: TGZ/GIT support for zlib, szip, libaec
## Testing Infrastructure
### Test Framework Structure
- **CTest integration**: Primary test runner
- **Express levels**: 0 (exhaustive) to 3 (quick) - default level 3
- **Timeout system**: Base 1200s, with short/long/very-long variants
- **Parallel testing**: Separate test infrastructure for MPI builds
### Test Categories
- **Core tests**: `test/` - Library unit tests
- **API tests**: `test/API/` - Comprehensive API validation
- **Parallel tests**: `testpar/` - MPI-specific functionality
- **Tool tests**: `tools/test/` - Command-line utility validation
- **VFD tests**: Multiple Virtual File Driver implementations
- **VOL tests**: Virtual Object Layer connector testing
- **Language binding tests**: C++, Fortran, Java specific tests
### Test Execution Patterns
```bash
# Express testing (quick)
export HDF_TEST_EXPRESS=3
ctest -j4
# VFD testing matrix
ctest -R "VFD"
# Parallel testing
ctest -R "MPI\|parallel"
# API comprehensive tests
ctest -R "H5_api_test"
```
## CI/CD Pipeline Architecture
### GitHub Actions Matrix
- Comprehensive testing across platforms/compilers
- **Multi-dimensional matrix**:
- Platforms: Windows (MSVC), Ubuntu (GCC), macOS (Clang)
- Features: Serial, Parallel, Thread-safe, Various language bindings
- Build types: Debug, Release, specialized configurations
### Key Workflow Categories
**Main CI Workflows:**
- `main.yml` - Primary CI across platforms
- `daily-build.yml` - Nightly comprehensive builds
- `ctest.yml` - Cross-platform testing with preset system
**Specialized Testing:**
- `par-*.yml` - Parallel/MPI testing workflows
- `vfd-*.yml` - Virtual File Driver testing
- `vol_*.yml` - Virtual Object Layer connector testing
- `analysis.yml` - Static analysis integration
**Platform-Specific:**
- `arm-main.yml` - ARM architecture testing
- `cygwin.yml`, `msys2.yml` - Windows alternative environments
- `intel.yml`, `aocc.yml`, `nvhpc.yml` - Vendor compiler support
**Release Infrastructure:**
- `release.yml` - Main release workflow with optional Maven deployment
- `release-files.yml` - Automated release packaging
- `tarball.yml` - Source distribution creation
- `maven-deploy.yml` - Maven repository deployment workflow
- `maven-staging.yml` - PR-based Maven testing and validation workflow
- `daily-schedule.yml` - Scheduled builds with AWS integration
### Build Matrix Strategy
- **Cross-platform validation**: Windows/Linux/macOS for every PR
- **Compiler diversity**: GCC, Clang, MSVC, Intel, AOCC, NVHPC
- **Feature combinations**: Systematic testing of feature interactions
- **Performance variants**: Debug vs Release vs specialized builds
## Packaging & Distribution
### Installation System
- **Component-based**: Libraries, headers, tools, docs, examples
- **CMake integration**: Full config package support
- **Cross-platform**: Windows (NSIS/WiX), macOS (DMG/Framework), Linux (DEB/RPM/TGZ)
### Release Process
- **Automated packaging**: CPack integration with platform-specific installers
- **Version management**: Automatic version extraction from source
- **SOVERSION handling**: Complex library versioning for different components
- **External dependencies**: Bundled compression libraries option
### Distribution Formats
- **Source tarballs**: Automated via GitHub Actions
- **Binary packages**: Platform-specific installers
- **Maven repositories**: GitHub Packages and Maven Central deployment
- **Java artifacts**: Platform-specific JARs with classifiers (linux-x86_64, windows-x86_64, macos-x86_64, macos-aarch64)
- **Container support**: Docker environments for CI
- **HPC integration**: Specialized configurations for batch systems
## Development Workflow Recommendations
### Build Strategy
1. **Use presets for consistency**: Leverage the preset system for reproducible builds
2. **Feature isolation**: Test individual features before combining
3. **Express testing**: Use level 3 for development, lower levels for validation
4. **Parallel development**: Separate MPI builds from serial development
### Testing Strategy
1. **Incremental testing**: Start with `testhdf5` core tests
2. **Feature-specific testing**: Use test regex patterns for targeted testing
3. **Cross-platform validation**: Test on primary CI platforms early
4. **Performance testing**: Include Release builds for performance-critical changes
### CI Integration
1. **Workflow triggers**: Understand which changes trigger which test suites
2. **Matrix optimization**: Consider CI time costs for comprehensive testing
3. **Failure isolation**: Use workflow categories to isolate platform/feature issues
4. **External dependency management**: Plan for compression library updates
### Maven Integration Workflow
1. **Java Build Configuration**: Enable Maven deployment with `HDF5_ENABLE_MAVEN_DEPLOY=ON`
2. **Version Management**: Use `HDF5_MAVEN_SNAPSHOT=ON` for development builds with `-SNAPSHOT` suffix
3. **Preset Selection**: Choose Maven-enabled presets (`ci-StdShar-GNUC-Maven` or `ci-StdShar-GNUC-Maven-Snapshot`)
4. **Platform Artifacts**: Automatic generation of platform-specific JARs with classifiers (linux-x86_64, windows-x86_64, macos-x86_64, macos-aarch64)
5. **CI Integration**: Conditional Maven artifact generation in `ctest.yml` workflow via preset system
6. **PR Testing**: Automated Maven artifact validation for pull requests via `maven-staging.yml`
7. **Validation Framework**: Pre-deployment validation using `.github/scripts/validate-maven-artifacts.sh`
8. **Repository Selection**: Choose between GitHub Packages and Maven Central via workflow inputs
9. **Release Integration**: Optional Maven deployment in release workflow with user control
## Critical Dependencies
### Build Dependencies
- **CMake 3.26+**: Required for preset support and modern features
- **Ninja**: Preferred generator for cross-platform consistency
- **Compression libraries**: zlib, szip/libaec (optional but commonly used)
- **MPI**: Required for parallel builds (MPI-3 standard minimum)
- **Java 11+**: Required for Java bindings and Maven deployment (when `HDF5_BUILD_JAVA=ON`)
- **Maven**: Optional for local Maven operations and validation
### Platform-Specific Requirements
- **Windows**: Visual Studio 2022, optional NSIS/WiX for packaging
- **macOS**: Xcode command line tools, universal binary support
- **Linux**: GCC/Clang, various package managers for dependencies
### Development Tools Integration
- **Static analysis**: Clang tools integration available
- **Code formatting**: clang-format integration
- **Coverage**: Code coverage support for testing
- **Sanitizers**: Runtime error detection support
- **Maven validation**: `.github/scripts/validate-maven-artifacts.sh` for pre-deployment validation
## Future Considerations
### Build System Evolution
- **CMake modernization**: Potential for newer CMake features as minimum version increases
- **Preset expansion**: More specialized presets for emerging use cases
- **Container integration**: Enhanced Docker/container support for development
### Testing Infrastructure
- **Test parallelization**: Opportunities for faster CI execution
- **Cloud testing**: Integration with cloud-native testing platforms
- **Performance regression**: Automated performance monitoring integration
### Platform Support
- **Emerging architectures**: ARM64, RISC-V support expansion
- **New compilers**: Integration with emerging compiler technologies
- **HPC evolution**: Adaptation to evolving supercomputing environments
Reference in New Issue View Git Blame Copy Permalink