When using the '-s' (h5ls) or '-r' (h5dump) option to display 1-byte integer datasets and attributes as ASCII characters, a closing double-quote character for data values was dropped in some cases.
A declared cd_nelmts with no values following it (e.g. UD=<filtn>,<flag>,1)
was never committed to filt->cd_nelmts: the trailing token (no comma after
it) was always treated as a value, never checked against the l/f/p pending
flags the way the comma-triggered branch does. h5repack's own validation
then silently passed since cd_nelmts stayed at its default 0, and the
mismatch was only ever caught by coincidence when the underlying filter
plugin did its own internal cd_values check (issue #6462).
Commit the trailing token to whichever UD field is still pending, mirroring
the comma-triggered branch exactly.
Co-authored-by: H. Joe Lee <hyoklee@hdfgroup.org>
* Updated h5dump --xml
The location of the files HDF5-File.xsd and HDF5-File.dtd no longer exists, and
the files were added to the repo (PR #5490). Before the --xml is actually removed,
running the h5dump tests will fail. This PR updated h5dump and the expected output
to use the correct files' location and actually completed the ticket HELP-2668.
* Added missing expected files from the previous commit
* Fixed typo
* Made corrections per review feedbacks
* Committing clang-format changes
* Missing files from previous commit
* Use valid URL for xmlns
---------
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
After PR #6352 made h5repack default to H5F_LIBVER_V18 as the low bound,
the existing metadata block size option check broke because with V18+
the larger meta_block_size shrinks the output file rather than growing
it. Loop the check over every H5F_LIBVER_* low bound and assert the
size ordering appropriate to each regime.
feat: add optional digital signature verification for HDF5 filter plugins
Introduce an opt-in plugin signing and verification system that allows
HDF5 deployments to require cryptographically signed filter plugins before
loading them. Disabled by default (HDF5_REQUIRE_SIGNED_PLUGINS=OFF).
New tool: h5sign
- Signs plugin shared libraries by appending an RSA signature and a
14-byte footer (algo_id | sig_len | 8-byte magic | format_ver) to the
binary without modifying the original content.
- Supports SHA-512 (default), SHA-256, SHA-384, and their PSS variants
(-a/--algorithm flag).
- Detects already-signed plugins; --force strips the old signature and
re-signs.
- Security hardened: keeps the file descriptor open through hashing and
appending (no TOCTOU window), enforces a 2048-bit minimum RSA key size,
rolls back partial writes on failure, and rejects paths that are not
regular files.
Verification (H5PLsig.c)
- At plugin load time, reads the footer, validates the magic and format
version, then checks the RSA signature against all public keys found in
the KeyStore directory.
- File is hashed once; per-key verification operates on the pre-computed
digest (no redundant I/O for multi-key keystores).
- Plugins whose signature hash appears in revoked_signatures.txt are
rejected regardless of key validity.
- Runtime debug output via HDF5_DEBUG=pl.
KeyStore management
- Trusted public keys are PEM files in a directory specified by
HDF5_PLUGIN_KEYSTORE_DIR (build time) or HDF5_PLUGIN_KEYSTORE (env var).
- HDF5_LOCK_PLUGIN_KEYSTORE cmake option disables the env-var override for
security-hardened deployments.
Test infrastructure
- h5signverifytest: positive, negative, tamper, re-sign, and revocation
test cases.
- CTest fixture-based dependency graph (FIXTURES_SETUP/FIXTURES_REQUIRED)
replaces fragile DEPENDS chains so tests remain correct under -R filtering.
- Dedicated signed-plugins.yml CI workflow; full test suite scoped to
H5SIGN and H5PLUGIN-signature tests to avoid unrelated flaky failures.
- Cross-platform: Linux, macOS, and Windows (MSVC-compatible, BIO-based
OpenSSL I/O, HDsleep/HDsetenv portability wrappers).
Documentation: docs/PLUGIN_SIGNATURE_README.md covers usage, footer
format, revocation file format, FAQ, and troubleshooting.
Co-authored-by: Glenn Song <gsong@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Re-write zlib/szip CMake logic for clarity
* Update external libaec, zlib-ng and zlib builds to not use patching process
* Add FindZLIBNG module to locate zlib-ng on system
* Rework HDF5 filter plugins support
Intel's icx compiler defines _WIN32 (with underscore) but not WIN32,
causing unistd.h and other POSIX headers to be incorrectly included
on Windows builds.
Refactor the cross-compilation infrastructure to rely on the standard
CMAKE_CROSSCOMPILING_EMULATOR variable instead of the custom
CROSSCOMPILING_PATH environment hack.
Key changes:
- Removed `ENVIRONMENT "${CROSSCOMPILING_PATH}"` from all test definitions.
- Updated custom commands (e.g., H5match_types, H5_gen) to invoke the
emulator explicitly when cross-compiling.
- Updated ConfigureChecks.cmake and HDF5UseFortran.cmake to enable
`try_run` checks when an emulator is defined.
- Added config/toolchain/mingw-w64-x86-64-wine.sh wrapper to automatically
locate MinGW runtime libraries (like libgfortran) for Wine execution.
- Cleaned up toolchain files to remove hardcoded emulator paths, deferring
configuration to the build environment.
Adds predefined datatypes for FP6 data in E2M3 and E3M2 formats
Does not add support for any native FP6 types; datatype conversions are performed in software
Adds predefined datatypes for FP8 data in E4M3 and E5M2 formats
Does not add support for any native FP8 types; datatype conversions are performed in software
Change default file format to 1.8 across various tests and examples, updating file creation and access logic accordingly.
Behavior:
Default file format version changed to 1.8 in H5Pfapl.c.
Updated file creation and access to use 1.8 format in h5ex_g_compact.c and test_file_image.c.
Set earliest file format in multiple test files including cache_tagging.c, dtypes.c, and links.c.
Tests:
Modified expected output in tools/test/misc/expected/*.ls files to reflect new file format locations.
Adjusted test logic in test_file_image.c and cache_tagging.c to accommodate format changes.
Misc:
Added comments and TODOs for future format testing in test_file_image.c.
Minor variable renaming for clarity in test_file_image.c.
Ensure proper closure of resources in gent_array1_big, gent_nested_compound_dt, and gent_intscalars in h5dumpgentest.c.
Add error handling and cleanup logic to prevent resource leaks.
Behavior:
Modify HDF5UseFortran.cmake to handle cross-compilation by providing default KINDs and sizes when CMAKE_CROSSCOMPILING is true.
Default values for INTEGER and REAL KINDs and their sizes are set when cross-compiling.
Error messages are adjusted to reflect cross-compilation scenarios.
Logic:
Conditional checks added for CMAKE_CROSSCOMPILING to determine if default values should be used.
Default values include INTEGER KINDs {1,2,4,8,16} and REAL KINDs {4,8,10,16}.
Default sizes for native kinds are set to 4 for INTEGER and REAL, and 8 for DOUBLE PRECISION when cross-compiling.
Misc:
Adjustments to verbose messages to indicate when defaults are used during cross-compilation.
Add VOL tests for h5dump, update test generation logic, and modify test configurations.
Tests:
Add VOL tests for h5dump in CMakeTests.cmake and CMakeTestsXML.cmake.
Modify ADD_H5_TEST macro to support VOL-specific tests and filtering.
Add gent_tvms() function in h5dumpgentest.c and h5dumpgentest.h for generating test files.
Adds predefined datatypes for little- and big-endian bfloat16 data
Does not add support for any native bfloat16 types; datatype conversions are performed in software
Also adds missing float16 predefined types to fortran
The trefer_extR test depends upon this pair of files, but always uses the pre-generated ones checked into the repo. This adds a routine to generate them, which will soon be used by the h5dump VOL tests.
Refactor ADD_H5_TEST macro in CMakeTests.cmake to consolidate test definitions and improve consistency.
Macros:
Consolidated ADD_H5_TEST and ADD_H5_ERR_TEST into a single ADD_H5_TEST macro in CMakeTests.cmake.
ADD_H5_TEST now accepts RESULT_CODE and optional RESULT_ERRCHECK as keyword arguments.
Test Definitions:
Updated all test invocations to use the new ADD_H5_TEST macro format with RESULT_CODE and RESULT_ERRCHECK where applicable.
Removed redundant ADD_H5_ERR_TEST macro usage.
Behavior:
Ensures consistent test setup and error handling across all h5ls tests.
Improve test file cleanup in h5repacktst.c by using h5_delete_test_file and avoiding deletion of default-driver exclusive files with non-default drivers.
Behavior:
Use h5_delete_test_file in h5repacktst.c for file deletions, ensuring compatibility with Family VFD filename requirements.
Avoid deleting default-driver exclusive files with non-default drivers in h5repacktst.c.
File Management:
Reorganize file arrays in h5repackgentest.h to separate default-driver files from others.
Adds workflow to build ROS3 VFD and optionally build aws-c-s3 library
from source or use package managers
Adds testing of ROS3 VFD with s3proxy and docker
Adds new H5Pset_fapl_ros3_endpoint()/H5Pget_fapl_ros3_endpoint() API
functions to set/get an alternative endpoint URL to use when opening
files with the ROS3 VFD
Cleans up warnings in tools and tests related to ROS3 VFD structure size
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Allen Byrne <50328838+byrnHDF@users.noreply.github.com>
jhendersonHDF
scivisionandscivision
Scot BreitenfeldandH. Joe Lee
Aleksandar Jelenak
Julien Schueller
Matt L
Neil Fortner
Allen Byrne
Dana Robinson