mirrors/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2026-09-25 04:09:44 +03:00
Code Issues Packages Projects Releases Wiki Activity

h5ls, h5dump, and h5stat don't need explicit ROS3 VFD option when S3 URI is used (#5684)

Browse Source 
* Tools assume ROS3 VFD when S3 URI is used
   h5dump, h5ls, and h5stat so far
* Add h5dump, h5ls, and h5stat tests
This commit is contained in:
Aleksandar Jelenak
2025-07-29 14:00:32 -05:00
committed by GitHub
parent fc3f153742
commit 21ab06b517
8 changed files with 82 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
release_docs/RELEASE.txt
+20 -12
View File
@@ -78,7 +78,7 @@ New Features
- Added CMAKE_INSTALL_PREFIX to the default plugin path
To help users find their plugins, the default plugin path has been
To help users find their plugins, the default plugin path has been
changed to include the CMAKE_INSTALL_PREFIX. Adding the install prefix
allows users to skip setting the HDF5_PLUGIN_PATH environment variable
when using plugins with the default lib/plugin location.
@@ -319,7 +319,7 @@ New Features
h5dump, h5ls and h5stat tools. However, this option has been kept for
compatibility reasons and can still be used to force the specified
credentials to take precedence over any credentials that the VFD would
otherwise try to source.
otherwise try to source.
Note that with these changes the AWS region to be used must always be
specified. However, the region can now be specified in several ways other
@@ -524,7 +524,7 @@ New Features
H5T_NATIVE_INT <-> H5T_NATIVE_FLOAT_COMPLEX | H5T_NATIVE_UINT <-> H5T_NATIVE_FLOAT_COMPLEX
H5T_NATIVE_LONG <-> H5T_NATIVE_FLOAT_COMPLEX | H5T_NATIVE_ULONG <-> H5T_NATIVE_FLOAT_COMPLEX
H5T_NATIVE_LLONG <-> H5T_NATIVE_FLOAT_COMPLEX | H5T_NATIVE_ULLONG <-> H5T_NATIVE_FLOAT_COMPLEX
H5T_NATIVE_FLOAT16 <-> H5T_NATIVE_FLOAT_COMPLEX | H5T_NATIVE_FLOAT <-> H5T_NATIVE_FLOAT_COMPLEX
H5T_NATIVE_FLOAT16 <-> H5T_NATIVE_FLOAT_COMPLEX | H5T_NATIVE_FLOAT <-> H5T_NATIVE_FLOAT_COMPLEX
H5T_NATIVE_DOUBLE <-> H5T_NATIVE_FLOAT_COMPLEX | H5T_NATIVE_LDOUBLE <-> H5T_NATIVE_FLOAT_COMPLEX
H5T_NATIVE_SCHAR <-> H5T_NATIVE_DOUBLE_COMPLEX | H5T_NATIVE_UCHAR <-> H5T_NATIVE_DOUBLE_COMPLEX
@@ -532,7 +532,7 @@ New Features
H5T_NATIVE_INT <-> H5T_NATIVE_DOUBLE_COMPLEX | H5T_NATIVE_UINT <-> H5T_NATIVE_DOUBLE_COMPLEX
H5T_NATIVE_LONG <-> H5T_NATIVE_DOUBLE_COMPLEX | H5T_NATIVE_ULONG <-> H5T_NATIVE_DOUBLE_COMPLEX
H5T_NATIVE_LLONG <-> H5T_NATIVE_DOUBLE_COMPLEX | H5T_NATIVE_ULLONG <-> H5T_NATIVE_DOUBLE_COMPLEX
H5T_NATIVE_FLOAT16 <-> H5T_NATIVE_DOUBLE_COMPLEX | H5T_NATIVE_FLOAT <-> H5T_NATIVE_DOUBLE_COMPLEX
H5T_NATIVE_FLOAT16 <-> H5T_NATIVE_DOUBLE_COMPLEX | H5T_NATIVE_FLOAT <-> H5T_NATIVE_DOUBLE_COMPLEX
H5T_NATIVE_DOUBLE <-> H5T_NATIVE_DOUBLE_COMPLEX | H5T_NATIVE_LDOUBLE <-> H5T_NATIVE_DOUBLE_COMPLEX
H5T_NATIVE_SCHAR <-> H5T_NATIVE_LDOUBLE_COMPLEX | H5T_NATIVE_UCHAR <-> H5T_NATIVE_LDOUBLE_COMPLEX
@@ -540,7 +540,7 @@ New Features
H5T_NATIVE_INT <-> H5T_NATIVE_LDOUBLE_COMPLEX | H5T_NATIVE_UINT <-> H5T_NATIVE_LDOUBLE_COMPLEX
H5T_NATIVE_LONG <-> H5T_NATIVE_LDOUBLE_COMPLEX | H5T_NATIVE_ULONG <-> H5T_NATIVE_LDOUBLE_COMPLEX
H5T_NATIVE_LLONG <-> H5T_NATIVE_LDOUBLE_COMPLEX | H5T_NATIVE_ULLONG <-> H5T_NATIVE_LDOUBLE_COMPLEX
H5T_NATIVE_FLOAT16 <-> H5T_NATIVE_LDOUBLE_COMPLEX | H5T_NATIVE_FLOAT <-> H5T_NATIVE_LDOUBLE_COMPLEX
H5T_NATIVE_FLOAT16 <-> H5T_NATIVE_LDOUBLE_COMPLEX | H5T_NATIVE_FLOAT <-> H5T_NATIVE_LDOUBLE_COMPLEX
H5T_NATIVE_DOUBLE <-> H5T_NATIVE_LDOUBLE_COMPLEX | H5T_NATIVE_LDOUBLE <-> H5T_NATIVE_LDOUBLE_COMPLEX
H5T_NATIVE_FLOAT_COMPLEX <-> H5T_NATIVE_DOUBLE_COMPLEX
@@ -628,11 +628,11 @@ New Features
There is no API compatibility wrapper for this change.
Fixes GitHub issue #3506
- H5Pset* routines now fail when used on default property lists
Modifying default property lists was never fully supported and could produce
inconsistent and unexpected behavior.
inconsistent and unexpected behavior.
- H5Pset_vol() now fails when used on a non-file-access property list
@@ -675,6 +675,14 @@ New Features
This option is supported by the following tools:
h5dump, h5ls, h5stat
- Specifying ROS3 VFD on the command line not required when using S3 URI
If using an S3 URI to reference HDF5 file in S3 (example:
s3://mybucket/myfile.h5) then ROS3 VFD will be automatically selected
unless the command-line option for the virtual file driver is used.
This feature applies to the following tools: h5dump, h5ls, h5stat.
- Deprecated h5dump XML option
The h5dump XML option is deprecated and will be removed in a future
@@ -786,7 +794,7 @@ Bug Fixes since HDF5-2.0.0 release
would cause those routines to return FAIL instead of FALSE when checking
the existence of a non-existent object with a file ID instead of a
group ID.
- Fixed a segfault in h5dump when a B-tree node level is corrupted
h5dump produced a segfault on a mal-formed file because a B-tree node
@@ -847,7 +855,7 @@ Bug Fixes since HDF5-2.0.0 release
Move the default setting of CMAKE_DEBUG_POSTFIX to the cacheinit.cmake file
usually used by testing. If CMAKE_DEBUG_POSTFIX is not set with a -D option
then CMAKE_DEBUG_POSTFIX will be the default provided by CMake itself.
- The relative rpaths ($ORIGIN / @loader_path) are appended to the CMAKE_INSTALL_RPATH
The RPATH settings were removed by a pull-request #5271, but the settings are needed
@@ -887,7 +895,7 @@ Bug Fixes since HDF5-2.0.0 release
be an error.
- Removed the module search find_package for szip library
There is not a szip module file to use, so the find_package only uses
find_package in config mode. The choice then is to either build szip, with libaec,
inline or find a system installed szip library, built with CMake.
@@ -950,7 +958,7 @@ Bug Fixes since HDF5-2.0.0 release
C++ APIs
--------
-
-
Testing
@@ -1007,7 +1015,7 @@ Platforms Tested
Linux 5.14.0-427.24.1.el9_4 GNU gcc, gfortran, g++ (Red Hat 11.4.1-3)
#1 SMP x86_64 GNU/Linux clang version 17.0.6
Rocky 9 Intel(R) oneAPI DPC++/C++ Compiler 2024.2.0
ifx (IFX) 2024.2.0
ifx (IFX) 2024.2.0
(cmake)
Linux-4.18.0-553.16.1.1toss.t4 openmpi/4.1.2
tools/lib/h5tools.h
+3
View File
@@ -739,6 +739,9 @@ H5TOOLS_DLL bool h5tools_render_region_element(FILE *stream, const h5tool_format
hsize_t *curr_pos, size_t ncols, hsize_t *ptdata,
hsize_t local_elmt_counter, hsize_t elmt_counter);
/* S3 schema prefix */
#define S3_URI_PREFIX "s3://"
#ifdef __cplusplus
}
#endif
tools/src/h5dump/h5dump.c
+18
View File
@@ -1471,6 +1471,24 @@ main(int argc, char *argv[])
while (H5_optind < argc) {
fname = strdup(argv[H5_optind++]);
if ((!use_custom_vfd_g) && (strncmp(fname, S3_URI_PREFIX, strlen(S3_URI_PREFIX)) == 0)) {
#ifdef H5_HAVE_ROS3_VFD
vfd_info_g.type = VFD_BY_NAME;
vfd_info_g.u.name = drivernames[ROS3_VFD_IDX];
vfd_info_g.info = ros3_fa_g;
use_custom_vfd_g = true;
if (h5tools_set_fapl_vfd(fapl_id, &vfd_info_g) < 0) {
error_msg("unable to set ROS3 VFD on fapl for file\n");
h5tools_setstatus(EXIT_FAILURE);
goto done;
}
#else
error_msg("ROS3 VFD is not available unless enabled when HDF5 is configured and built.\n");
h5tools_setstatus(EXIT_FAILURE);
goto done;
#endif
}
/* A short cut to get the revision count of an onion file without opening the file */
if (get_onion_revision_count && H5FD_ONION == H5Pget_driver(fapl_id)) {
uint64_t revision_count = 0;
tools/src/h5ls/h5ls.c
+20
View File
@@ -3159,6 +3159,26 @@ main(int argc, char *argv[])
oname = NULL;
file_id = H5I_INVALID_HID;
/* Since this tool does not handle different VFD per input file the code below is not
* changing much. Once a file triggers custom_vfd_fapl boolean to true, ROS3 will apply
* to any subsequent file. */
if ((!custom_vfd_fapl) && (strncmp(fname, S3_URI_PREFIX, strlen(S3_URI_PREFIX)) == 0)) {
#ifdef H5_HAVE_ROS3_VFD
vfd_info.type = VFD_BY_NAME;
vfd_info.u.name = drivernames[ROS3_VFD_IDX];
custom_vfd_fapl = true;
vfd_info.info = ros3_fa;
if (h5tools_set_fapl_vfd(fapl_id, &vfd_info) < 0) {
error_msg("unable to set ROS3 VFD on fapl for file\n");
leave(EXIT_FAILURE);
}
#else
error_msg(
"Error: ROS3 VFD is not available unless enabled when HDF5 is configured and built.\n\n");
leave(EXIT_FAILURE);
#endif
}
while (fname && *fname) {
file_id = h5tools_fopen(fname, H5F_ACC_RDONLY, fapl_id, (custom_vol_fapl || custom_vfd_fapl),
drivername, sizeof drivername);
tools/src/h5stat/h5stat.c
+18
View File
@@ -1804,6 +1804,24 @@ main(int argc, char *argv[])
hid_t fcpl;
H5F_info2_t finfo;
if ((!use_custom_vfd_g) && (strncmp(fname, S3_URI_PREFIX, strlen(S3_URI_PREFIX)) == 0)) {
#ifdef H5_HAVE_ROS3_VFD
vfd_info_g.type = VFD_BY_NAME;
vfd_info_g.u.name = drivernames[ROS3_VFD_IDX];
vfd_info_g.info = ros3_fa_g;
use_custom_vfd_g = true;
if (h5tools_set_fapl_vfd(fapl_id, &vfd_info_g) < 0) {
error_msg("unable to set ROS3 VFD on fapl for file\n");
h5tools_setstatus(EXIT_FAILURE);
goto done;
}
#else
error_msg("ROS3 VFD is not available unless enabled when HDF5 is configured and built.\n");
h5tools_setstatus(EXIT_FAILURE);
goto done;
#endif
}
fprintf(rawoutstream, "Filename: %s\n", fname);
fid = h5tools_fopen(fname, H5F_ACC_RDONLY, fapl_id, (use_custom_vol_g || use_custom_vfd_g), NULL, 0);
tools/test/h5dump/CMakeTests.cmake
+1 -1
View File
@@ -1567,5 +1567,5 @@ if (HDF5_ENABLE_ROS3_VFD_DOCKER_PROXY)
ADD_H5_S3TEST (tattrintsize 0 anon s3 h5dumpros3 --filedriver=ros3 --vfd-name=ros3 --s3-cred=\(,,\) --endpoint-url=http://localhost:${h5dump_s3tests_port})
ADD_H5_S3TEST (tattrintsize 0 profile http localhost:${h5dump_s3tests_port}/h5dumpros3 --vfd-name=ros3)
ADD_H5_S3TEST (tattrintsize 0 profile s3 h5dumpros3 --filedriver=ros3 --vfd-name=ros3 --endpoint-url=http://localhost:${h5dump_s3tests_port})
ADD_H5_S3TEST (tattrintsize 0 filename s3 h5dumpros3 --endpoint-url=http://localhost:${h5dump_s3tests_port})
endif ()
tools/test/h5ls/CMakeTests.cmake
+1
View File
@@ -577,6 +577,7 @@ if (HDF5_ENABLE_ROS3_VFD_DOCKER_PROXY)
ADD_H5_S3TEST (tgroup 0 anon s3 h5lsros3 --vfd-name=ros3 --s3-cred=\(,,\) --endpoint-url=http://localhost:${h5ls_s3tests_port})
ADD_H5_S3TEST (tgroup 0 profile http localhost:${h5ls_s3tests_port}/h5lsros3 --vfd-name=ros3)
ADD_H5_S3TEST (tgroup 0 profile s3 h5lsros3 --vfd-name=ros3 --endpoint-url=http://localhost:${h5ls_s3tests_port})
ADD_H5_S3TEST (tgroup 0 filename s3 h5lsros3 --endpoint-url=http://localhost:${h5ls_s3tests_port})
endif ()
##############################################################################
tools/test/h5stat/CMakeTests.cmake
+1
View File
@@ -356,4 +356,5 @@ if (HDF5_ENABLE_ROS3_VFD_DOCKER_PROXY)
ADD_H5_S3TEST (h5stat_threshold 0 anon s3 h5statros3 --vfd-name=ros3 --s3-cred=\(,,\) --endpoint-url=http://localhost:${h5stat_s3tests_port})
ADD_H5_S3TEST (h5stat_threshold 0 profile http localhost:${h5stat_s3tests_port}/h5statros3 --vfd-name=ros3)
ADD_H5_S3TEST (h5stat_threshold 0 profile s3 h5statros3 --vfd-name=ros3 --endpoint-url=http://localhost:${h5stat_s3tests_port})
ADD_H5_S3TEST (h5stat_threshold 0 filename s3 h5statros3 --endpoint-url=http://localhost:${h5stat_s3tests_port})
endif ()