From 508695977838cff9f4f4f7ce3d5ff92d0ca765e2 Mon Sep 17 00:00:00 2001 From: Larry Knox Date: Thu, 25 Apr 2024 13:33:26 -0500 Subject: [PATCH] Sync develop changes April 4 -April 22 to hdf5_1_14 (#4437) * Fixed dead links in docs (#4334) * Revert "Fixed dead links in docs (#4334)" (#4335) This reverts commit 408fc6600292e945c8df32802eb9b9241093ce75. * Fix grammar and content in HDF5Examples (#4333) * Remove HD macros for C time functions (#4337) * HDasctime * HDclock * HDctime * HDdifftime * HDgmtime * HDlocaltime * HDmktime * HDtime * HDtzset HDgettimeofday will be done later * Remove HD prefix from toupper/tolower (#4340) * Add VDS and SWMR to documentation (#4336) * Clean up random number generator code (#4338) * Clean up random number generator code Depending on the platform, we use a mix of random, rand, and rand_r to generate pseudo-random numbers, along with a messy set of ifdefs in H5private.h. We are not a cryptographic library, only use random numbers in our test code, and have no need for anything more than the C standard's (s)rand(). There's no point dithering about using rand() vs random() when we're also doing bad things like using mod to restrict the range, which introduces bias. Also removes CMake/configure checks for rand_r and random * Remove random/rand_r checks from build system * Fix missed HDrandom after GitHub merge * Remove the getting started guide (#4339) This was moved to the wiki * Remove noise comment from H5DO.c (#4342) * Add note about H5Iregister_type hash_table param (#4345) The hash_table parameter is unused in HDF5 1.8.13 and later, when we moved from using hash tables to skip lists for storing IDs. The current hash table implementation does not use this parameter. * Remove `#undef ERR` from H5private.h (#4346) This is an old work-around that is no longer necessary. The only place where we defined ERR is in h5import, and that code has been updated (in this PR) to rename ERR to INVALID_TOKEN. * Add NAMECHECK_ONLY to macro list in .clang-format (#4343) This caused H5TS.c functions to be mis-formatted * Remove HD from last non-win32 C89 functions (#4348) Removes the HD prefix from the last C89 functions with no special Windows equilvalent: * rename * rewind * setbuf * signal * tmpfile * Fix dead links cont. (#4349) Added img/images_to_copy.dox as a temporary solution because doxygen didn't copy the images used in the examples/*.html files - will investigate more. This was necessary for the links to intro_SWMR.html and intro_VDS.html. * Fix a typo (#4352) * Fix use of possibly uninitialized memory in h5repack (#4354) * Match content listing with content. (#4357) * Remove obsolete c++ doxygen config files (#4358) * Fix typo VERSIO (#4363) * Removed duplicate entries in compile line for h5* compiler wrappers (#4360) * h5fc remove duplicate args * h5 wrappers remove duplicate args * Added CVE info to RELEASE.txt (#4367) * Change doxygen version from 1.10.0 to 1.9.7 (#4371) * Fix all doxygen versions (#4377) * Change date to data in INSTALL_parallel (#4384) * Fix a broken link for zlib url (#4383) * Fix broken links and references in INSTALL_Auto.txt (#4382) * Remove signature in comment (#4366) * switched from \returns to commonly used \return (#4389) * Add CXX flags (#4359) * Remove TRACE macros (#4341) * Set available CMake build types for non-multi-config generators (#4330) * Correct the high-level library link (#4398) * Correct the high-level library link The link currently used points to 1.13 release. Changed to use the md file instead. * Update Doxyfile for md files * Update to include md files * Add additional terms to the glossary (#4401) Fixes GH #4390 * added missing doxygen \since (#4410) * Update Intel oneAPI version to 2024.1 (#4412) * Update documentation (#4403) - Added File Image Operations to Technical Notes * Use a correct example location * Use aliases for URLs (#4409) * Fix CXX sanitizer by moving marco include to after CXX code. (#4414) * Update and use more URL alias (#4413) * Update documentation (#4415) Added the replacement for https://docs.hdfgroup.org/archive/support/HDF5/doc/TechNotes/Version.html Corrected some minor typos and another deadlink * Remove obsolete file (#4416) * Update oneAPI version to 2024.1 for Daily Build (#4418) * Add training videos page (#4422) * Updates for release_docs/NEWSLETTER.txt. (#4423) * Add release process markdown file (#4312) * Lowercase in File Format Specification (#4424) * Fixing string in RELEASE_PROCESS.md that is causing a doxygen error - (#4430) Unsupported xml/html tag found. * Allow clang-format to format H5Cpkg.h (#4429) Co-authored-by: Larry Knox * Fix broken links in md files needing full URLs. * Add URL for Java examples that was accidentally dropped. Change paths from develop to 1.14 directories for some doxygen generated files. * Correct typo. * Fix 2 more dead URLs. * Revert "Clean up random number generator code (#4338)" This reverts commit 58e4d0842a9ab1c65722f672cabc38da22d48bd8. --- .clang-format | 4 +- .github/workflows/cmake-ctest.yml | 4 +- .github/workflows/intel-auto.yml | 2 +- .github/workflows/intel-cmake.yml | 4 +- CMakeLists.txt | 72 +- CONTRIBUTING.md | 2 +- README.md | 6 +- autogen.sh | 5 +- bin/README.md | 2 +- bin/release | 2 +- bin/trace | 450 +-- config/cmake/HDFMacros.cmake | 3 + config/sanitizer/sanitizers.cmake | 45 + config/toolchain/clang.cmake | 1 + doc/getting-started-with-hdf5-development.md | 866 ----- doc/parallel-compression.md | 28 +- doxygen/Doxyfile.in | 2 +- doxygen/aliases | 229 +- doxygen/dox/About.dox | 16 +- doxygen/dox/ExamplesAPI.dox | 621 ++-- doxygen/dox/GettingStarted.dox | 4 +- doxygen/dox/Glossary.dox | 17 +- doxygen/dox/IntroHDF5.dox | 7 +- doxygen/dox/IntroParExamples.dox | 16 +- doxygen/dox/IntroParHDF5.dox | 9 +- doxygen/dox/LearnBasics.dox | 20 +- doxygen/dox/LearnBasics2.dox | 2 +- doxygen/dox/LearnBasics3.dox | 28 +- doxygen/dox/LearnHDFView.dox | 18 +- doxygen/dox/Overview.dox | 10 +- doxygen/dox/TechnicalNotes.dox | 15 + doxygen/dox/TrainingVideos.dox | 48 + doxygen/dox/UsersGuide.dox | 22 +- doxygen/dox/VOLConnGuide.dox | 4 +- doxygen/dox/ViewTools.dox | 55 +- doxygen/dox/ViewToolsJPSS.dox | 4 +- doxygen/dox/rm-template.dox | 4 +- doxygen/examples/FileFormat.html | 2 +- doxygen/examples/FileImageOps.html | 1610 +++++++++ doxygen/examples/H5.format.html | 6 +- .../LibraryReleaseVersionNumbers.html | 318 ++ doxygen/examples/intro_SWMR.html | 4 +- doxygen/hdf5doxy_layout.xml | 4 +- doxygen/img/images_to_copy.dox | 1 + doxygen/img/trunk_branches.jpg | Bin 0 -> 51564 bytes fortran/src/README.md | 2 +- hl/src/H5DO.c | 14 - hl/src/H5DOpublic.h | 2 +- hl/src/H5LTpublic.h | 6 +- hl/tools/h5watch/h5watch.c | 4 +- java/src/hdf/overview.html | 2 +- release_docs/INSTALL | 2 +- release_docs/INSTALL_Auto.txt | 30 +- release_docs/INSTALL_CMake.txt | 2 +- release_docs/INSTALL_parallel | 4 +- release_docs/NEWSLETTER.txt | 13 +- release_docs/README_HDF5_CMake | 25 - release_docs/RELEASE_PROCESS.md | 163 + src/H5.c | 15 - src/H5A.c | 44 - src/H5Adeprec.c | 5 - src/H5Apublic.h | 7 +- src/H5Clog_json.c | 46 +- src/H5Clog_trace.c | 2 +- src/H5Cpkg.h | 3050 ++++++++--------- src/H5D.c | 46 - src/H5Ddbg.c | 1 - src/H5Ddeprec.c | 4 - src/H5Dmpio.c | 4 +- src/H5Dpublic.h | 15 +- src/H5E.c | 20 - src/H5ES.c | 14 - src/H5ESpublic.h | 24 +- src/H5Edeprec.c | 8 - src/H5Epublic.h | 6 + src/H5F.c | 47 - src/H5FD.c | 37 - src/H5FDcore.c | 12 +- src/H5FDdirect.c | 2 - src/H5FDfamily.c | 2 - src/H5FDhdfs.c | 2 - src/H5FDlog.c | 9 +- src/H5FDmirror.c | 2 - src/H5FDmpio.c | 8 - src/H5FDmulti.c | 6 - src/H5FDonion.c | 7 +- src/H5FDperform.c | 1 - src/H5FDros3.c | 4 - src/H5FDs3comms.c | 10 +- src/H5FDsec2.c | 9 +- src/H5FDsplitter.c | 2 - src/H5FDstdio.c | 2 - src/H5FDsubfiling/H5FDsubfiling.c | 4 - src/H5FDwindows.c | 1 - src/H5Fdeprec.c | 3 - src/H5Fmodule.h | 8 +- src/H5Fmpi.c | 2 - src/H5Fpublic.h | 22 +- src/H5G.c | 18 - src/H5Gdeprec.c | 15 - src/H5I.c | 23 +- src/H5Ipublic.h | 5 +- src/H5L.c | 32 - src/H5Ldeprec.c | 6 - src/H5M.c | 23 - src/H5Mpublic.h | 28 +- src/H5O.c | 39 - src/H5Odbg.c | 8 +- src/H5Odeprec.c | 11 - src/H5Omtime.c | 4 +- src/H5Opublic.h | 4 + src/H5P.c | 26 - src/H5PL.c | 9 - src/H5Pdapl.c | 12 - src/H5Pdcpl.c | 28 - src/H5Pdeprec.c | 8 - src/H5Pdxpl.c | 27 - src/H5Pfapl.c | 60 - src/H5Pfcpl.c | 18 - src/H5Pgcpl.c | 8 - src/H5Plapl.c | 10 - src/H5Plcpl.c | 2 - src/H5Pmapl.c | 2 - src/H5Pocpl.c | 19 - src/H5Pocpypl.c | 6 - src/H5Ppublic.h | 29 +- src/H5Pstrcpl.c | 2 - src/H5R.c | 17 - src/H5Rdeprec.c | 7 - src/H5S.c | 15 - src/H5Sall.c | 1 - src/H5Sdeprec.c | 1 - src/H5Shyper.c | 8 - src/H5Snone.c | 1 - src/H5Spoint.c | 3 - src/H5Spublic.h | 2 + src/H5Sselect.c | 14 - src/H5T.c | 20 - src/H5TS.c | 9 +- src/H5Tarray.c | 5 - src/H5Tcommit.c | 10 - src/H5Tcompound.c | 5 - src/H5Tconv.c | 12 +- src/H5Tcset.c | 2 - src/H5Tdeprec.c | 2 - src/H5Tenum.c | 5 - src/H5Tfields.c | 3 - src/H5Tfixed.c | 2 - src/H5Tfloat.c | 8 - src/H5Tnative.c | 1 - src/H5Toffset.c | 2 - src/H5Topaque.c | 2 - src/H5Torder.c | 2 - src/H5Tpad.c | 2 - src/H5Tprecis.c | 2 - src/H5Tpublic.h | 3 +- src/H5Tstrpad.c | 2 - src/H5Tvlen.c | 1 - src/H5VL.c | 27 - src/H5VLcallback.c | 95 - src/H5VLmodule.h | 2 +- src/H5VLnative.c | 2 - src/H5Z.c | 4 - src/H5module.h | 10 +- src/H5private.h | 156 +- src/H5public.h | 4 +- src/H5system.c | 10 +- src/H5timer.c | 6 +- test/API/H5_api_test.c | 2 +- test/API/README.md | 4 +- test/accum.c | 2 +- test/app_ref.c | 4 +- test/big.c | 2 +- test/btree2.c | 6 +- test/cmpd_dset.c | 2 +- test/del_many_dense_attrs.c | 2 +- test/dsets.c | 2 +- test/dt_arith.c | 4 +- test/dtypes.c | 2 +- test/earray.c | 2 +- test/farray.c | 2 +- test/fheap.c | 6 +- test/filenotclosed.c | 2 +- test/h5test.c | 14 +- test/hyperslab.c | 2 +- test/istore.c | 2 +- test/links.c | 8 +- test/mdset.c | 2 +- test/mtime.c | 8 +- test/set_extent.c | 2 +- test/swmr_reader.c | 4 +- test/swmr_remove_reader.c | 4 +- test/swmr_sparse_reader.c | 4 +- test/tcheck_version.c | 4 +- test/tid.c | 2 +- test/tselect.c | 18 +- test/tskiplist.c | 2 +- test/ttime.c | 4 +- test/tunicode.c | 2 +- test/unlink.c | 2 +- test/vds_swmr_writer.c | 4 +- testpar/API/H5_api_test_parallel.c | 2 +- testpar/t_2Gio.c | 4 +- testpar/t_pmulti_dset.c | 2 +- testpar/t_shapesame.c | 4 +- testpar/testphdf5.c | 4 +- tools/lib/h5tools_utils.c | 2 +- tools/src/h5import/h5import.c | 10 +- tools/src/h5import/h5import.h | 52 +- tools/src/h5ls/h5ls.c | 4 +- 210 files changed, 4762 insertions(+), 4795 deletions(-) delete mode 100644 doc/getting-started-with-hdf5-development.md create mode 100644 doxygen/dox/TrainingVideos.dox create mode 100644 doxygen/examples/FileImageOps.html create mode 100644 doxygen/examples/LibraryReleaseVersionNumbers.html create mode 100644 doxygen/img/trunk_branches.jpg delete mode 100644 release_docs/README_HDF5_CMake create mode 100644 release_docs/RELEASE_PROCESS.md diff --git a/.clang-format b/.clang-format index 0b6225579c0..59a9bd479ae 100644 --- a/.clang-format +++ b/.clang-format @@ -11,7 +11,7 @@ BraceWrapping: BreakBeforeBraces: Stroustrup BreakAfterJavaFieldAnnotations: true BreakStringLiterals: true -ColumnLimit: 110 # Update $max_trace_macro_line_len in bin/trace also +ColumnLimit: 110 IndentWidth: 4 --- Language: Cpp @@ -68,12 +68,14 @@ StatementMacros: - CATCH - END_MEMBERS - FUNC_ENTER_API + - FUNC_ENTER_API_NAMECHECK_ONLY - FUNC_ENTER_NOAPI - FUNC_ENTER_NOAPI_NOERR - FUNC_ENTER_NOAPI_NOINIT - FUNC_ENTER_NOAPI_NOINIT_NOERR - FUNC_ENTER_PACKAGE - FUNC_LEAVE_API + - FUNC_LEAVE_API_NAMECHECK_ONLY - FUNC_LEAVE_NOAPI - FUNC_LEAVE_NOAPI_NOFS - H5E_END_TRY diff --git a/.github/workflows/cmake-ctest.yml b/.github/workflows/cmake-ctest.yml index 5315f56e1c4..19197ae5762 100644 --- a/.github/workflows/cmake-ctest.yml +++ b/.github/workflows/cmake-ctest.yml @@ -411,7 +411,7 @@ jobs: id: setup-fortran with: compiler: intel - version: '2023.2' + version: '2024.1' - name: Enable Developer Command Prompt uses: ilammy/msvc-dev-cmd@v1.13.0 @@ -510,7 +510,7 @@ jobs: id: setup-fortran with: compiler: intel - version: '2023.2' + version: '2024.1' - name: Set file base name (Linux_intel) id: set-file-base diff --git a/.github/workflows/intel-auto.yml b/.github/workflows/intel-auto.yml index fa3e17573b3..129a7bec6cb 100644 --- a/.github/workflows/intel-auto.yml +++ b/.github/workflows/intel-auto.yml @@ -30,7 +30,7 @@ jobs: id: setup-fortran with: compiler: intel - version: '2024.0' + version: '2024.1' - name: Autotools Configure shell: bash diff --git a/.github/workflows/intel-cmake.yml b/.github/workflows/intel-cmake.yml index b1338444dde..5492c9967be 100644 --- a/.github/workflows/intel-cmake.yml +++ b/.github/workflows/intel-cmake.yml @@ -31,7 +31,7 @@ jobs: id: setup-fortran with: compiler: intel - version: '2024.0' + version: '2024.1' - name: CMake Configure (Linux) shell: bash @@ -85,7 +85,7 @@ jobs: id: setup-fortran with: compiler: intel - version: '2024.0' + version: '2024.1' - name: CMake Configure (Windows) shell: pwsh diff --git a/CMakeLists.txt b/CMakeLists.txt index ee65c2a607d..ec1ecae58ef 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -533,42 +533,6 @@ if (BUILD_STATIC_EXECS) endif () endif () -option (HDF5_ENABLE_ANALYZER_TOOLS "enable the use of Clang tools" OFF) -mark_as_advanced (HDF5_ENABLE_ANALYZER_TOOLS) -if (HDF5_ENABLE_ANALYZER_TOOLS) - include (${HDF5_SOURCE_DIR}/config/sanitizer/tools.cmake) -endif () -option (HDF5_ENABLE_SANITIZERS "execute the Clang sanitizer" OFF) -mark_as_advanced (HDF5_ENABLE_SANITIZERS) -if (HDF5_ENABLE_SANITIZERS) - include (${HDF5_SOURCE_DIR}/config/sanitizer/sanitizers.cmake) -endif () -option (HDF5_ENABLE_FORMATTERS "format source files" OFF) -mark_as_advanced (HDF5_ENABLE_FORMATTERS) -if (HDF5_ENABLE_FORMATTERS) - include (${HDF5_SOURCE_DIR}/config/sanitizer/formatting.cmake) -endif () - -#----------------------------------------------------------------------------- -# Option to use code coverage -#----------------------------------------------------------------------------- -option (HDF5_ENABLE_COVERAGE "Enable code coverage for Libraries and Programs" OFF) -if (HDF5_ENABLE_COVERAGE) - include (${HDF5_SOURCE_DIR}/config/sanitizer/code-coverage.cmake) - if(CODE_COVERAGE AND CODE_COVERAGE_ADDED) - add_code_coverage () # Adds instrumentation to all targets - else () - set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -O0 --coverage -fprofile-arcs -ftest-coverage") - set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g --coverage -O0 -fprofile-arcs -ftest-coverage") - if (CMAKE_C_COMPILER_ID STREQUAL "GNU") - set (LDFLAGS "${LDFLAGS} -fprofile-arcs -ftest-coverage") - link_libraries (gcov) - else () - set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} --coverage") - endif () - endif () -endif () - #----------------------------------------------------------------------------- # Option to indicate using a memory checker #----------------------------------------------------------------------------- @@ -1175,4 +1139,40 @@ if (EXISTS "${HDF5_SOURCE_DIR}/HDF5Examples" AND IS_DIRECTORY "${HDF5_SOURCE_DIR endif () endif () +option (HDF5_ENABLE_ANALYZER_TOOLS "enable the use of Clang tools" OFF) +mark_as_advanced (HDF5_ENABLE_ANALYZER_TOOLS) +if (HDF5_ENABLE_ANALYZER_TOOLS) + include (${HDF5_SOURCE_DIR}/config/sanitizer/tools.cmake) +endif () +option (HDF5_ENABLE_SANITIZERS "execute the Clang sanitizer" OFF) +mark_as_advanced (HDF5_ENABLE_SANITIZERS) +if (HDF5_ENABLE_SANITIZERS) + include (${HDF5_SOURCE_DIR}/config/sanitizer/sanitizers.cmake) +endif () +option (HDF5_ENABLE_FORMATTERS "format source files" OFF) +mark_as_advanced (HDF5_ENABLE_FORMATTERS) +if (HDF5_ENABLE_FORMATTERS) + include (${HDF5_SOURCE_DIR}/config/sanitizer/formatting.cmake) +endif () + +#----------------------------------------------------------------------------- +# Option to use code coverage +#----------------------------------------------------------------------------- +option (HDF5_ENABLE_COVERAGE "Enable code coverage for Libraries and Programs" OFF) +if (HDF5_ENABLE_COVERAGE) + include (${HDF5_SOURCE_DIR}/config/sanitizer/code-coverage.cmake) + if(CODE_COVERAGE AND CODE_COVERAGE_ADDED) + add_code_coverage () # Adds instrumentation to all targets + else () + set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -O0 --coverage -fprofile-arcs -ftest-coverage") + set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g --coverage -O0 -fprofile-arcs -ftest-coverage") + if (CMAKE_C_COMPILER_ID STREQUAL "GNU") + set (LDFLAGS "${LDFLAGS} -fprofile-arcs -ftest-coverage") + link_libraries (gcov) + else () + set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} --coverage") + endif () + endif () +endif () + include (CMakeInstallation.cmake) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3ffb0509eba..1f58a5be16d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -122,7 +122,7 @@ Please make sure that you check the items applicable to your pull request: * [ ] If changes were done to Autotools build, were they added to CMake and vice versa? * [ ] Is the pull request applicable to any other branches? If yes, which ones? Please document it in the GitHub issue. * [ ] Is the new code sufficiently documented for future maintenance? - * [ ] Does the new feature require a change to an existing API? See "API Compatibility Macros" document (https://docs.hdfgroup.org/hdf5/v1_14/api-compat-macros.html) + * [ ] Does the new feature require a change to an existing API? See "API Compatibility Macros" document (https://hdfgroup.github.io/hdf5/v1_14/api-compat-macros.html) * Documentation * [ ] Was the change described in the release_docs/RELEASE.txt file? * [ ] Was the new function documented in the corresponding public header file using [Doxygen](https://hdfgroup.github.io/hdf5/v1_14/_r_m_t.html)? diff --git a/README.md b/README.md index 98f093f83d2..ee65b1e1399 100644 --- a/README.md +++ b/README.md @@ -21,15 +21,15 @@ DOCUMENTATION ------------- This release is fully functional for the API described in the documentation. - https://docs.hdfgroup.org/hdf5/v1_14/_l_b_a_p_i.html + [HDF5 C API](https://hdfgroup.github.io/hdf5/v1_14/_l_b_a_p_i.html) Full Documentation and Programming Resources for this release can be found at - https://docs.hdfgroup.org/hdf5/v1_14/index.html + [Full Documentation](https://hdfgroup.github.io/hdf5/v1_14/index.html) The latest doxygen documentation generated on changes to HDF5 1.14.x is available at: - https://hdfgroup.github.io/hdf5/ + https://hdfgroup.github.io/hdf5/v1_14 See the [RELEASE.txt](release_docs/RELEASE.txt) file in the [release_docs/](release_docs/) directory for information specific to the features and updates included in this release of the library. diff --git a/autogen.sh b/autogen.sh index 33a82779465..31d117cf8ed 100755 --- a/autogen.sh +++ b/autogen.sh @@ -44,9 +44,8 @@ echo # compilation. # Run trace script -# The trace script adds H5TRACE macros to library source files. It should -# have no effect on files that don't have HDF5 API macros in them. -echo "Running trace script:" +# The trace script updates H5ARG_TRACE macros in library source files. +echo "Running arg trace script:" bin/trace src/H5*.c || exit 1 echo diff --git a/bin/README.md b/bin/README.md index e2e25701893..28559f0ba60 100644 --- a/bin/README.md +++ b/bin/README.md @@ -21,7 +21,7 @@ Programs run via `autogen.sh` (or the equivalent in CMake) are indicated. |`restore.sh`|Removes files generated by `autogen.sh`| |`runbkprog`|Used by CMake to run test programs in the background| |`switch_maint_mode`|Switches maintainer mode on/off in `configure.ac`| -|`trace`|Adds `TRACE` macros to HDF5 C library source files (run by `autogen.sh`)| +|`trace`|Updates `H5ARG_TRACE` macros in H5ES\_insert() calls (run by `autogen.sh`)| |`warnhist`|Generates compiler warning statistics for gcc/clang when fed output of make| ## TODO diff --git a/bin/release b/bin/release index 25b450efb86..f72390cda08 100755 --- a/bin/release +++ b/bin/release @@ -455,7 +455,7 @@ cp -p Makefile.dist Makefile # Update README.md and release_docs/RELEASE.txt with release information in # line 1. -for f in README.md release_docs/RELEASE.txt; do +for f in README.md release_docs/RELEASE.txt release_docs/NEWSLETTER.txt; do echo "HDF5 version $VERS released on $release_date" >$f.x sed -e 1d $f >>$f.x mv $f.x $f diff --git a/bin/trace b/bin/trace index 5941f3a89a8..f86b2ee7874 100755 --- a/bin/trace +++ b/bin/trace @@ -10,6 +10,12 @@ # If you do not have access to either file, you may request a copy from # help@hdfgroup.org. ## + +## +# Process H5ARG_TRACE() macros (defined in H5private.h) used in H5ES_insert(), +# updating them with the caller's parameters, etc. +## + require 5.003; use warnings; $Source = ""; @@ -26,6 +32,10 @@ $Source = ""; # Two-character strings begin with an upper-case letter which is # usually the same as the package name. # +# These map to the types in H5trace.c (which should be updated if you add +# a type here), so there are more types listed here than are used by the +# existing macros. +# %TypeString = ("haddr_t" => "a", "H5A_info_t" => "Ai", "H5A_operator1_t" => "Ao", @@ -210,294 +220,180 @@ $Source = ""; "H5VL_request_status_t" => "#", ); - -############################################################################## -# Maximum length of H5TRACE macro line -# If the ColumnLimit in .clang-format is changed, this value will need to be updated -# -my $max_trace_macro_line_len = 110; - - ############################################################################## # Print an error message. # my $found_errors = 0; sub errmesg ($$@) { - my ($file, $func, @mesg) = @_; - my ($mesg) = join "", @mesg; - my ($lineno) = 1; - if ($Source =~ /(.*?\n)($func)/s) { - local $_ = $1; - $lineno = tr/\n/\n/; - } + my ($file, $func, @mesg) = @_; + my ($mesg) = join "", @mesg; + my ($lineno) = 1; + if ($Source =~ /(.*?\n)($func)/s) { + local $_ = $1; + $lineno = tr/\n/\n/; + } - $found_errors = 1; + $found_errors = 1; - print "$file: in function \`$func\':\n"; - print "$file:$lineno: $mesg\n"; + print "$file: in function \`$func\':\n"; + print "$file:$lineno: $mesg\n"; } ############################################################################## -# Given a C data type return the type string that goes with it. +# Given a C data type, return the type string that goes with it # sub argstring ($$$) { - my ($file, $func, $atype) = @_; - my ($ptr, $tstr, $array) = (0, "!", ""); - my ($fq_atype); + my ($file, $func, $atype) = @_; + my ($ptr, $tstr, $array) = (0, "!", ""); + my ($fq_atype); - # Normalize the data type by removing redundant white space, - # certain type qualifiers, and indirection. - $atype =~ s/^\bconst\b//; # Leading const - $atype =~ s/\s*const\s*//; # const after type, possibly in the middle of '*'s - $atype =~ s/^\bstatic\b//; - $atype =~ s/\bH5_ATTR_UNUSED\b//g; - $atype =~ s/\bH5_ATTR_DEPRECATED_USED\b//g; - $atype =~ s/\bH5_ATTR_NDEBUG_UNUSED\b//g; - $atype =~ s/\bH5_ATTR_DEBUG_API_USED\b//g; - $atype =~ s/\bH5_ATTR_PARALLEL_UNUSED\b//g; - $atype =~ s/\bH5_ATTR_PARALLEL_USED\b//g; - $atype =~ s/\s+/ /g; - $ptr = length $1 if $atype =~ s/(\*+)//; - $atype =~ s/^\s+//; - $atype =~ s/\s+$//; - if ($atype =~ /(.*)\[(.*)\]$/) { - ($array, $atype) = ($2, $1); + # Normalize the data type by removing redundant white space, + # certain type qualifiers, and indirection. + $atype =~ s/^\bconst\b//; # Leading const + $atype =~ s/\s*const\s*//; # const after type, possibly in the middle of '*'s + $atype =~ s/^\bstatic\b//; + $atype =~ s/\bH5_ATTR_UNUSED\b//g; + $atype =~ s/\bH5_ATTR_DEPRECATED_USED\b//g; + $atype =~ s/\bH5_ATTR_NDEBUG_UNUSED\b//g; + $atype =~ s/\bH5_ATTR_PARALLEL_UNUSED\b//g; + $atype =~ s/\bH5_ATTR_PARALLEL_USED\b//g; + $atype =~ s/\s+/ /g; + $ptr = length $1 if $atype =~ s/(\*+)//; + $atype =~ s/^\s+//; $atype =~ s/\s+$//; - } - $fq_atype = $atype . ('*' x $ptr); + if ($atype =~ /(.*)\[(.*)\]$/) { + ($array, $atype) = ($2, $1); + $atype =~ s/\s+$//; + } + $fq_atype = $atype . ('*' x $ptr); - if ($ptr>0 && exists $TypeString{$fq_atype}) { - $ptr = 0; - $tstr = $TypeString{$fq_atype}; - } elsif ($ptr>0 && exists $TypeString{"$atype*"}) { - --$ptr; - $tstr = $TypeString{"$atype*"}; - } elsif (!exists $TypeString{$atype}) { -# Defer throwing error until type is actually used -# errmesg $file, $func, "untraceable type \`$atype", '*'x$ptr, "\'"; - } else { - $tstr = $TypeString{$atype}; - } - return ("*" x $ptr) . ($array ? "[$array]" : "") . $tstr; + if ($ptr>0 && exists $TypeString{$fq_atype}) { + $ptr = 0; + $tstr = $TypeString{$fq_atype}; + } elsif ($ptr>0 && exists $TypeString{"$atype*"}) { + --$ptr; + $tstr = $TypeString{"$atype*"}; + } elsif (!exists $TypeString{$atype}) { + # Defer throwing error until type is actually used + # errmesg $file, $func, "untraceable type \`$atype", '*'x$ptr, "\'"; + } else { + $tstr = $TypeString{$atype}; + } + return ("*" x $ptr) . ($array ? "[$array]" : "") . $tstr; } ############################################################################## # Given information about an API function, rewrite that function with # updated tracing information. # -my $file_api = 0; my $file_args = 0; -my $total_api = 0; my $total_args = 0; sub rewrite_func ($$$$$) { - my ($file, $type, $name, $args, $body) = @_; - my ($arg, $trace, $argtrace); - my (@arg_name, @arg_str, @arg_type); - local $_; + my ($file, $type, $name, $args, $body) = @_; + my ($arg, $argtrace); + my (@arg_name, @arg_str, @arg_type); + local $_; - # Keep copy of original arguments - my $orig_args = $args; + # Keep copy of original arguments + my $orig_args = $args; - # Parse return value - my $rettype = argstring $file, $name, $type; + # Parse arguments + if ($args eq "void") { + $argtrace = "H5ARG_TRACE0(\"\")"; + } else { + # Split arguments + # + # First remove: + # * /*in*/, /*out*/, /*in_out*/, and /*in,out*/ comments + # * preprocessor lines that start with # + # * H5FL_TRACKING_PARAMS macros (free list code only) + # + # then split the function arguments on commas + $args =~ s/\/\*\s*in\s*\*\///g; # Get rid of /*in*/ + $args =~ s/\/\*\s*out\s*\*\///g; # Get rid of /*out*/ + $args =~ s/\/\*\s*in,\s*out\s*\*\///g; # Get rid of /*in,out*/ + $args =~ s/\/\*\s*in_out\s*\*\///g; # Get rid of /*in_out*/ + $args =~ s/H5FL_TRACK_PARAMS//g; # Remove free list macro + $args =~ s/\n#.*?\n/\n/g; # Remove lines beginning with '#' + my @args = split /,[\s\n]*/, $args; + my $argno = 0; + my %names; - # Parse arguments - if ($args eq "void") { - $trace = "H5TRACE0(\"$rettype\", \"\");\n"; - $argtrace = "H5ARG_TRACE0(\"\")"; - } else { - # Split arguments - # - # First remove: - # * /*in*/, /*out*/, /*in_out*/, and /*in,out*/ comments - # * preprocessor lines that start with # - # * H5FL_TRACKING_PARAMS macros (free list code only) - # - # then split the function arguments on commas - $args =~ s/\/\*\s*in\s*\*\///g; # Get rid of /*in*/ - $args =~ s/\/\*\s*out\s*\*\///g; # Get rid of /*out*/ - $args =~ s/\/\*\s*in,\s*out\s*\*\///g; # Get rid of /*in,out*/ - $args =~ s/\/\*\s*in_out\s*\*\///g; # Get rid of /*in_out*/ - $args =~ s/H5FL_TRACK_PARAMS//g; # Remove free list macro - $args =~ s/\n#.*?\n/\n/g; # Remove lines beginning with '#' - my @args = split /,[\s\n]*/, $args; - my $argno = 0; - my %names; + for $arg (@args) { + if($arg=~/\w*\.{3}\w*/){ # Skip "..." for varargs parameter + next; + } + unless ($arg=~/^((\s*[a-z_A-Z](\w|\*)*\s+)+(\s*\*\s*|\s*const\s*|\s*restrict\s*|\s*volatile\s*)*) + ([a-z_A-Z]\w*)(\[.*?\])?\s*$/x) { + errmesg $file, $name, "unable to parse \`$arg\'"; + goto error; + } else { + my ($atype, $aname, $array, $adir) = ($1, $5, $6, $8); - for $arg (@args) { - if($arg=~/\w*\.{3}\w*/){ # Skip "..." for varargs parameter - next; - } - unless ($arg=~/^((\s*[a-z_A-Z](\w|\*)*\s+)+(\s*\*\s*|\s*const\s*|\s*volatile\s*)*) - ([a-z_A-Z]\w*)(\[.*?\])?\s*$/x) { - errmesg $file, $name, "unable to parse \`$arg\'"; - goto error; - } else { - my ($atype, $aname, $array, $adir) = ($1, $5, $6, $8); - $names{$aname} = $argno++; - $adir ||= "in"; - $atype =~ s/\s+$//; - push @arg_name, $aname; - push @arg_type, $atype; + $names{$aname} = $argno++; + $adir ||= "in"; + $atype =~ s/\s+$//; + push @arg_name, $aname; + push @arg_type, $atype; - if ($adir eq "out") { - push @arg_str, "x"; - } else { - if (defined $array) { - $atype .= "*"; - if ($array =~ /^\[\/\*([a-z_A-Z]\w*)\*\/\]$/) { - my $asize = $1; - if (exists $names{$asize}) { - $atype .= '[a' . $names{$asize} . ']'; - } else { - warn "bad array size: $asize"; - $atype .= "*"; - } + if ($adir eq "out") { + push @arg_str, "x"; + } else { + if (defined $array) { + $atype .= "*"; + if ($array =~ /^\[\/\*([a-z_A-Z]\w*)\*\/\]$/) { + my $asize = $1; + + if (exists $names{$asize}) { + $atype .= '[a' . $names{$asize} . ']'; + } else { + warn "bad array size: $asize"; + $atype .= "*"; + } + } + } + + push @arg_str, argstring $file, $name, $atype; + } } - } - push @arg_str, argstring $file, $name, $atype; } - } + + # Compose the trace macro + $argtrace = "H5ARG_TRACE" . scalar(@arg_str) . "(__func__, \""; + $argtrace .= join("", @arg_str) . "\""; + + # Append arguments + for my $i (0 .. $#arg_name) { + $argtrace .= ", $arg_name[$i]"; + } + + # Append final ')' for macro + $argtrace .= ")"; } - # Compose the trace macro - $trace = "H5TRACE" . scalar(@arg_str) . "(\"$rettype\", \""; - $argtrace = "H5ARG_TRACE" . scalar(@arg_str) . "(__func__, \""; - $trace .= join("", @arg_str) . "\""; - $argtrace .= join("", @arg_str) . "\""; + # Check for H5ARG_TRACE macros to update + if ( $body =~ /H5ARG_TRACE/ ) { + my $orig_body = $body; - # Add 4 for indenting the line - my $len = 4 + length($trace); - - for my $i (0 .. $#arg_name) { - # Handle wrapping - - # Be VERY careful here! clang-format and this script MUST agree - # on which lines get wrapped or there will be churn as each tries - # to undo the other's output. - # - # TWO cases must be handled: - # 1) The argument is that last one and ');' will be appended - # 2) The argument is NOT the last one and ',' will be appended - # - # NB: clang-format does NOT consider terminal newlines when - # counting columns for the ColumnLimit - # - # The extra '2' added after $len includes the ', ' that would be - # added BEFORE the argument. - # - my $adjust = ($i + 1 == scalar(@arg_str)) ? 2 : 1; - my $len_if_added = $len + 2 + length($arg_name[$i]) + $adjust; - - # Wrap lines that will be longer than the limit - if ($len_if_added > $max_trace_macro_line_len) { - # Wrap line, with indentation - $trace .= ",\n "; - $len = 13; # Set to 13, for indentation - - # Indent an extra space to account for extra digit in 'H5TRACE' macro - if (scalar(@arg_str) >= 10) { - $trace .= " "; - $len++; + # Check for untraceable type (deferred until $argtrace used) + if ( $argtrace =~ /(^!)|([^*]!)/ ) { + errmesg $file, $name, "untraceable type in args"; + print "args = '$orig_args'\n"; + goto error; } - } else { - $trace .= ", "; - $len += 2; # Add 2, for ', ' - } - # Append argument - $trace .= "$arg_name[$i]"; - $argtrace .= ", $arg_name[$i]"; + # Update H5ARG_TRACE macro + $body =~ s/(H5ARG_TRACE(\d+\s*\(.*?\))?)/"$argtrace"/esg; - # Add length of appended argument name - $len += length($arg_name[$i]); + # Increment # of non-API routines modified if anything changed + if ($orig_body ne $body) { + $file_args++; + } } - # Append final ');' for macro - $trace .= ");\n"; - $argtrace .= ")"; - } - - # Check for API / non-API routine name - if( $name =~ /H5[A-Z]{0,2}[a-z].*/) { - # The H5TRACE() statement, for API routines - if ($body =~ /\/\*[ \t]*NO[ \t]*TRACE[ \t]*\*\//) { - # Ignored due to NO TRACE comment. - } else { - # Check for known, but unsupported type - if ( $trace =~ /(^#)|([^*]#)/ ) { - # Check for unsupported return type - if ( $type =~ /(^#)|([^*]#)/ ) { - errmesg $file, $name, "unsupported type in return type\nAdd to TypeString hash in trace script and update H5_trace_args()"; - print "type = '$type'\n"; - } - - # Check for unsupported argument type - $index = 0; - for (@arg_str) { - if ( $_ =~ /(^#)|([^*]#)/ ) { - errmesg $file, $name, "unsupported type in args\nAdd to TypeString hash in trace script and update H5_trace_args()"; - print "type = $arg_type[$index]\n"; - } - $index++; - } - goto error; - } - - # Check for unknown (and therefore unsupported) type - if ( $trace =~ /(^!)|([^*]!)/ ) { - # Check for unsupported return type - if ( $type =~ /(^!)|([^*]!)/ ) { - errmesg $file, $name, "unknown type in return type\nAdd to TypeString hash in trace script and also update H5_trace_args() if used by value"; - print "type = '$type'\n"; - } - - # Check for unsupported argument type - $index = 0; - for (@arg_str) { - if ( $_ =~ /(^!)|([^*]!)/ ) { - errmesg $file, $name, "unknown type in args\nAdd to TypeString hash in trace script and also update H5_trace_args() if used by value"; - print "type = $arg_type[$index]\n"; - } - $index++; - } - goto error; - } - - if ($body =~ s/((\n[ \t]*)H5TRACE\d+\s*\(.*?\);)\n/"$2$trace"/es) { - # Replaced an H5TRACE macro. - } elsif ($body=~s/((\n[ \t]*)FUNC_ENTER\w*[ \t]*(\(.*?\))?;??)\n/"$1$2$trace"/es) { - # Added an H5TRACE macro after a FUNC_ENTER macro. - } else { - errmesg $file, $name, "unable to insert tracing information"; - print "body = ", $body, "\n"; - goto error; - } - } - - #Increment # of API routines modified - $file_api++; - } - - # Check for H5ARG_TRACE macros in non-API routines - if ( $body =~ /H5ARG_TRACE/ ) { - # Check for untraceable type (deferred until $argtrace used) - if ( $argtrace =~ /(^!)|([^*]!)/ ) { - errmesg $file, $name, "untraceable type in args"; - print "args = '$orig_args'\n"; - goto error; - } - - # Replace / update H5ARG_TRACE macro. - $body =~ s/(H5ARG_TRACE(\d+\s*\(.*?\))?)/"$argtrace"/esg; - - #Increment # of non-API routines modified - $file_args++; - } - - error: - return "\n$type\n$name($orig_args)\n$body"; +error: + return "\n$type\n$name($orig_args)\n$body"; } ############################################################################## @@ -505,41 +401,38 @@ sub rewrite_func ($$$$$) { # tracing information. # for $file (@ARGV) { - $file_api = 0; - $file_args = 0; + $file_args = 0; - # Ignore some files that do not need tracing macros - unless ($file eq "H5FDmulti.c" or $file eq "src/H5FDmulti.c" or $file eq "H5FDstdio.c" or $file eq "src/H5FDstdio.c" or $file eq "src/H5TS.c" or $file eq "src/H5FDperform.c") { + # Ignore the "external" source files that don't include H5private.h + unless ($file eq "H5FDmulti.c" or $file eq "src/H5FDmulti.c" or $file eq "H5FDstdio.c" or $file eq "src/H5FDstdio.c") { - # Snarf up the entire file - open SOURCE, $file or die "$file: $!\n"; - $Source = join "", ; - close SOURCE; + # Snarf up the entire file + open SOURCE, $file or die "$file: $!\n"; + $Source = join "", ; + close SOURCE; - # Make a copy of the original data - my $original = $Source; + # Make a copy of the original data + my $original = $Source; - # Make modifications - $Source =~ s/\n([A-Za-z]\w*(\s+[A-Za-z]\w*)*\s*\**)\n #type - (H5[A-Z]{0,2}_?[a-zA-Z0-9_]\w*) #name - \s*\((.*?)\)\s* #args - (\{.*?\n\}[^\n]*) #body - /rewrite_func($file,$1,$3,$4,$5)/segx; + # Make modifications + $Source =~ s/\n([A-Za-z]\w*(\s+[A-Za-z]\w*)*\s*\**)\n #type + (H5[A-Z]{0,2}_?[a-zA-Z0-9_]\w*) #name + \s*\((.*?)\)\s* #args + (\{.*?\n\}[^\n]*) #body + /rewrite_func($file,$1,$3,$4,$5)/segx; -# If the source changed then print out the new version - if ($original ne $Source) { - printf "%s: instrumented %d API function%s and %d argument list%s\n", - $file, $file_api, (1 == $file_api ? "" : "s"), - $file_args, (1 == $file_args ? "" : "s"); - rename $file, "$file~" or die "unable to make backup"; - open SOURCE, ">$file" or die "unable to modify source"; - print SOURCE $Source; - close SOURCE; + # If the source changed then print out the new version + if ($original ne $Source) { + printf "%s: Instrumented %d argument list%s\n", + $file, $file_args, (1 == $file_args ? "" : "s"); + rename $file, "$file~" or die "unable to make backup"; + open SOURCE, ">$file" or die "unable to modify source"; + print SOURCE $Source; + close SOURCE; - $total_api += $file_api; - $total_args += $file_args; + $total_args += $file_args; + } } - } } if ($found_errors eq 1) { @@ -549,9 +442,8 @@ if ($found_errors eq 1) { printf "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n"; exit 1; } else { - printf "Finished processing HDF5 API calls:\n"; - printf "\tinstrumented %d API function%s and %d argument list%s\n", - $total_api, (1 == $total_api ? "" : "s"), + printf "Finished processing H5ES_insert() calls:\n"; + printf "\tInstrumented %d argument list%s\n", $total_args, (1 == $total_args ? "" : "s"); } diff --git a/config/cmake/HDFMacros.cmake b/config/cmake/HDFMacros.cmake index 15338272ccf..cefdaa03863 100644 --- a/config/cmake/HDFMacros.cmake +++ b/config/cmake/HDFMacros.cmake @@ -79,6 +79,9 @@ macro (SET_HDF_BUILD_TYPE) else () set (HDF_CFG_NAME "Release") endif () + # Set available build types for cmake-gui/ccmake convenience + set_property (CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS + "Debug" "Release" "RelWithDebInfo" "MinSizeRel" "Developer") endif () endmacro () diff --git a/config/sanitizer/sanitizers.cmake b/config/sanitizer/sanitizers.cmake index 72f101f4c25..91768240fb2 100644 --- a/config/sanitizer/sanitizers.cmake +++ b/config/sanitizer/sanitizers.cmake @@ -43,11 +43,17 @@ function(test_san_flags return_var flags) set(FLAGS_BACKUP ${CMAKE_REQUIRED_FLAGS}) set(CMAKE_REQUIRED_FLAGS "${flags}") check_c_source_compiles("int main() { return 0; }" ${return_var}) + if(CMAKE_CXX_COMPILER_LOADED) + check_cxx_source_compiles("int main() { return 0; }" ${return_var}) + endif() set(CMAKE_REQUIRED_FLAGS "${FLAGS_BACKUP}") set(CMAKE_REQUIRED_QUIET "${QUIET_BACKUP}") endfunction() message(STATUS "USE_SANITIZER=${USE_SANITIZER}, CMAKE_C_COMPILER_ID=${CMAKE_C_COMPILER_ID}") +if(CMAKE_CXX_COMPILER_LOADED) + message(STATUS "... CMAKE_CXX_COMPILER_ID=${CMAKE_CXX_COMPILER_ID}") +endif() if(USE_SANITIZER) if(CMAKE_C_COMPILER_ID MATCHES "IntelLLVM" OR CMAKE_C_COMPILER_ID MATCHES "[Cc]lang") set(CMAKE_EXPORT_COMPILE_COMMANDS ON) @@ -79,9 +85,15 @@ if(USE_SANITIZER) if(AFL) append_quoteless(AFL_USE_ASAN=1 CMAKE_C_COMPILER_LAUNCHER) + if (CMAKE_CXX_COMPILER_LOADED) + append_quoteless(AFL_USE_ASAN=1 CMAKE_CXX_COMPILER_LAUNCHER) + endif () endif() else() message(FATAL_ERROR "Address sanitizer not available for ${CMAKE_C_COMPILER}") + if (CMAKE_CXX_COMPILER_LOADED) + message(FATAL_ERROR "Address sanitizer not available for ${CMAKE_CXX_COMPILER}") + endif () endif() endif() @@ -104,9 +116,15 @@ if(USE_SANITIZER) if(AFL) append_quoteless(AFL_USE_MSAN=1 CMAKE_C_COMPILER_LAUNCHER) + if (CMAKE_CXX_COMPILER_LOADED) + append_quoteless(AFL_USE_MSAN=1 CMAKE_CXX_COMPILER_LAUNCHER) + endif () endif() else() message(FATAL_ERROR "Memory [With Origins] sanitizer not available for ${CMAKE_C_COMPILER}") + if (CMAKE_CXX_COMPILER_LOADED) + message(FATAL_ERROR "Memory [With Origins] sanitizer not available for ${CMAKE_CXX_COMPILER}") + endif () endif() endif() @@ -123,9 +141,15 @@ if(USE_SANITIZER) if(AFL) append_quoteless(AFL_USE_UBSAN=1 CMAKE_C_COMPILER_LAUNCHER) + if (CMAKE_CXX_COMPILER_LOADED) + append_quoteless(AFL_USE_UBSAN=1 CMAKE_CXX_COMPILER_LAUNCHER) + endif () endif() else() message(FATAL_ERROR "Undefined Behaviour sanitizer not available for ${CMAKE_C_COMPILER}") + if (CMAKE_CXX_COMPILER_LOADED) + message(FATAL_ERROR "Undefined Behaviour sanitizer not available for ${CMAKE_CXX_COMPILER}") + endif () endif() endif() @@ -139,9 +163,15 @@ if(USE_SANITIZER) if(AFL) append_quoteless(AFL_USE_TSAN=1 CMAKE_C_COMPILER_LAUNCHER) + if (CMAKE_CXX_COMPILER_LOADED) + append_quoteless(AFL_USE_TSAN=1 CMAKE_CXX_COMPILER_LAUNCHER) + endif () endif() else() message(FATAL_ERROR "Thread sanitizer not available for ${CMAKE_C_COMPILER}") + if (CMAKE_CXX_COMPILER_LOADED) + message(FATAL_ERROR "Thread sanitizer not available for ${CMAKE_CXX_COMPILER}") + endif () endif() endif() @@ -155,9 +185,15 @@ if(USE_SANITIZER) if(AFL) append_quoteless(AFL_USE_LSAN=1 CMAKE_C_COMPILER_LAUNCHER) + if (CMAKE_CXX_COMPILER_LOADED) + append_quoteless(AFL_USE_LSAN=1 CMAKE_CXX_COMPILER_LAUNCHER) + endif () endif() else() message(FATAL_ERROR "Thread sanitizer not available for ${CMAKE_C_COMPILER}") + if (CMAKE_CXX_COMPILER_LOADED) + message(FATAL_ERROR "Thread sanitizer not available for ${CMAKE_CXX_COMPILER}") + endif () endif() endif() @@ -171,9 +207,15 @@ if(USE_SANITIZER) if(AFL) append_quoteless(AFL_USE_CFISAN=1 CMAKE_C_COMPILER_LAUNCHER) + if (CMAKE_CXX_COMPILER_LOADED) + append_quoteless(AFL_USE_CFISAN=1 CMAKE_CXX_COMPILER_LAUNCHER) + endif () endif() else() message(FATAL_ERROR "Control Flow Integrity(CFI) sanitizer not available for ${CMAKE_C_COMPILER}") + if (CMAKE_CXX_COMPILER_LOADED) + message(FATAL_ERROR "Control Flow Integrity(CFI) sanitizer not available for ${CMAKE_CXX_COMPILER}") + endif () endif() endif() @@ -198,6 +240,9 @@ if(USE_SANITIZER) if(AFL) append_quoteless(AFL_USE_ASAN=1 CMAKE_C_COMPILER_LAUNCHER) + if (CMAKE_CXX_COMPILER_LOADED) + append_quoteless(AFL_USE_ASAN=1 CMAKE_CXX_COMPILER_LAUNCHER) + endif () endif() else() message(FATAL_ERROR "This sanitizer not yet supported in the MSVC environment: ${USE_SANITIZER}") diff --git a/config/toolchain/clang.cmake b/config/toolchain/clang.cmake index 2d35641a14c..2a856077f11 100644 --- a/config/toolchain/clang.cmake +++ b/config/toolchain/clang.cmake @@ -9,6 +9,7 @@ if(WIN32) else() set (CMAKE_C_COMPILER clang) set (CMAKE_CXX_COMPILER clang++) + #set (CMAKE_Fortran_COMPILER flang-new) endif() set (CMAKE_EXPORT_COMPILE_COMMANDS ON) diff --git a/doc/getting-started-with-hdf5-development.md b/doc/getting-started-with-hdf5-development.md deleted file mode 100644 index 687a1ea7cca..00000000000 --- a/doc/getting-started-with-hdf5-development.md +++ /dev/null @@ -1,866 +0,0 @@ -# Getting Started with HDF5 Development - -## Introduction - -The purpose of this document is to introduce new HDF5 developers to some of the -quirks of our source code. It's not a style guide (see the forthcoming HDF5 -style guide for that), but instead describes the most commonly encountered -features that are likely to trip up someone who has never worked with the HDF5 -source code before. - -Corrections and suggestions for improvement should be handled via GitHub pull -requests and issues. - -## Getting started - -### Building the library for development - -You don't really need special configuration settings for building the library -as a developer. - -Some tips that may be useful: - -* Building in debug mode will turn on additional checks in many packages. - You'll probably want to start coding in debug mode. -* You can turn symbols on independently of debug/production mode. -* If you will be looking at memory issues via tools like valgrind, you will - need to turn off the free lists, which recycle memory so we can avoid - calling malloc/free. This is done using the `--enable-using-memchecker` - configure option. Some developers build with this all the time, as the - memory recyclilng can hide problems like use-after-free. -* You can enable developer warnings via `--enable-developer-warnings`. These - warnings generate a lot of noise, but the output can occasionally be useful. - I probably wouldn't turn them on all the time, though, as they can make it - harder to spot warnings that we care about. -* You can set warnings as errors. We have an older scheme that does this for - a subset of errors or you can simply specify `-Werror`, etc. as a part of - `CFLAGS`. Configure is smart enough to strip it out when it runs configure - checks. We build the C library with -Werror on GitHub, so you'll need to fix - your warnings before creating a pull request. -* CMake has a developer mode that turns most these settings on. - - -### Branches - -Please see `doc/branches-explained.md` for an explanation of our branching strategy. - -For new small features, we have developers create feature branches in their own -repositories and then create pull requests into the canonical HDF5 repository. -For larger work, especially when the work will be done by multiple people, we -create feature branches named `feature/`. If work stops on a feature -branch, we rename it to `inactive/`. - -If you create a feature branch in the canonical HDFGroup repository, please -create a `BRANCH.md` text file in the repository root and explain: - -* The branch's purpose -* Contact info for someone who can tell us about the branch -* Clues about when the branch will be merged or can be considered for retirement - -The purpose of this document is to avoid orphan branches with no clear -provenance. - - -### Pull requests - -The process of creating a pull request is explained in `CONTRIBUTING.md`. - - -## A brief tour of the source code - -Here's a quick guide to where you can find things in our source tree. Some of these directories have README.md files of their own. - -`bin/` -Scripts we use for building the software and misc. tools. - -`c++/` -Source, tests, and examples for the C++ language wrapper. - -`config/` -Configuration files for both the Autotools and CMake. - -`doc/` -Miscellaneous documents, mostly in markdown format. - -`doxygen/` -Mainly Doxygen build files and other top-level Doxygen things. The Doxygen content is spread across the library's header files but some content can be found here when it has no other obvious home. - -`examples/` -C library examples. Fortran and C++ examples are located in their corresponding wrapper directory. - -`fortran/` -Source, tests, and examples for the Fortran language wrapper. - -`hl/` -Source, tests, and examples for the high-level library. - -`java/` -Source, tests, and examples for the JNI language wrapper and the corresponding OO Java library. - -`m4/` -m4 build scripts used by the Autotools. CMake ignores these. - -`release_docs/` -Install instructions and release notes. - -`src/` -Source code for the C library. - -`test/` -C library test code. Described in much more detail below. - -`testpar/` -Parallel C library test code. Described in much more detail below. - -`tools/` -HDF5 command-line tools code, associated tools tests, and the input test files. - -`utils/` -Small utility programs that don't belong anywhere else. - - -## General Things - -### Necessary software - -In order to do development work on the HDF5 library, you will need to have -a few things available. - -* A C99-compatible C compiler (MSVC counts). C11 is required to build the subfiling feature. -* Either CMake or the Autotools (Autoconf, Automake, libtool) -* Perl is needed to run some scripts, even on Windows -* A C++11-compatible compiler if you want to build the C++ wrappers -* A Fortran 2003-compatible compiler if you want to build the Fortran wrappers -* A Java 8-compatible compiler if you want to build the Java wrappers -* flex/lex and bison/yacc if you want to modify the high-level parsers -* A development version of zlib is necessary for zlib compression -* A development version of szip is necessary for szip compression -* An MPI-3 compatible MPI library must be installed for parallel HDF5 development -* clang-format is handy for formatting your code before submission to GitHub. The formatter will automatically update your PR if it's mis-formatted, though, so this isn't strictly necessary. -* codespell is useful to identify spelling issues before submission to GitHub. The codespell action won't automatically correct your code, but it will point out spelling errors, so this also isn't strictly necessary. - -These are the requirements for working on the develop branch. Maintenance -branches may relax the required versions somewhat. For example, HDF5 1.12 and -earlier only require C++98. - -Certain optional features may require additional libraries to be installed. You'll need curl and some S3 components installed to build the read-only S3 VFD, for example. - -### Platform-independence - -HDF5 assumes you have a C99 compiler and, to a certain extent, a POSIX-like -environment (other languages will be discussed later). On most operating systems -in common use, this will be a reasonable assumption. The biggest exception to -this has been Windows, which, until recently, had poor C99 compliance and spotty -POSIX functionality. To work around differences in platforms and compilers, -we've implemented a compatibility scheme. - -Unlike most codebases, which test for features and inject their own normal-looking -functions when there are deficiencies, HDF5 uses a scheme where we prefix all -C and POSIX calls with `HD` (e.g., `HDmalloc`). The `H5private.h` header handles -most of the fixup for Unix-like operating systems and defines the HD replacements. -For Windows, we first parse the `H5win32defs.h` file, which maps missing Windows -and MSVC functionality to POSIX and C99 calls. `H5private.h` tests for previously -defined HD calls and skips redefining it if it already exists. H5system.c -includes Windows glue code as well as a few functions we need to paper over -differences in Unix-like systems. - -One thing to keep in mind when looking at our platform-independence layer is -that it is quite old, having been authored when the Unix world was much more -fragmented and C99 was uncommon. We've slowly been reworking it as POSIX has -standardized and C99 become widespread. - -Another thing to keep in mind is that we're fairly conservative about deprecating -support for older platforms and compilers. There's an awful lot of ancient -hardware that requires HDF5, so we try to only make major changes to things -like language requirements when we increment the major version (minor version -prior to HDF5 2.0). - -### C99 - -We assume you have a C99 compiler. Subfiling uses some C11 features, but that is -compiled on demand and we are not moving that requirement to the rest of the -library. All modern compilers we've tested can handle HDF5's C99 requirements, -even Microsoft's. In the future, we'll remove the `HD` prefixes from all standard -C library calls. - -One quirk of HDF5's age, is the `hbool_t` type, which was created before C99 -Booleans were widespread and which uses `TRUE` and `FALSE` macros for its values -instead of C99's `true` and `false`. We plan to switch this over to C99's Boolean -types sometime in the near future. - -### POSIX - -We assume basic POSIX.1-2008 functionality is present. When a POSIX (or common Unix) -function is missing on a popular platform, we implement a shim or macro -in `H5private.h` and/or `H5system.c`. Systems that need a lot of help, like -Windows, have gotten special headers in the past (e.g., `H5win32defs.h`) but -now that most platforms implement the POSIX and C99 functionality we need, these -special headers are less necessary. - -### Threads - -Thread-safety was originally implemented using Pthreads, with Win32 support -bolted on later. No other thread libraries are supported. The subfiling -feature uses multiple threads under the hood, but that's out of scope for -an introductory document. Thread-related code is largely confined to the `H5TS` -files, where we define HDF5-specific primitives and then map Pthread or Win32 -implementations onto them. - -### C++ - -The C++ Wrappers require C++11. We generally only require the rule of three -for the classes. - -### Fortran - -The Fortran wrappers require Fortran 2003. - -### Java - -The Java wrappers require Java 8. - -### Warning suppression - -In the rare cases where we've decided to suppress a warning, we have a set -of function-like macros that we use for that. They are located in `H5private.h` -and have the form `H5__DIAG_(OFF|ON)` and take the name of the warning -they are suppressing as a parameter. They were originally designed for gcc and -extended to clang. They have not been updated for other compilers. Instead, -we have plans to revamp the macro system to be more generic and extensible. - -We try to configure the compilers we use the most for maximum grumpiness and -then fix all the warnings we can. Please don't use the warning suppression -macros in lieu of actually fixing the underlying problems. - -## Build Systems - -We support building the library with both the Autotools and CMake. We'd like to -eventually move to only having one build system, which would be CMake since -the Autotools don't really support Windows, but that seems unlikely to happen -anytime soon. With few exceptions, any new feature, test, or configure -option should be supported in both build systems. - -The particulars of the build systems can be found in the `config` directory -and its subdirectories. - -## Working in the library - -### Anatomy of an HDF5 API call - -HDF5 API calls have a uniform structure imposed by our function enter/leave and -error handling schemes. We currently stick to this boilerplate for ALL -functions, though this may change in the future. The general boilerplate varies -slightly between internal and public API calls. - -Here's an example of an internal API call: - -```c -/* - * Function comments of dubious value - */ -herr_t -H5X_do_stuff(/*parameters*/) -{ - /* variables go here */ - void *foo = NULL; - herr_t ret_value = SUCCEED; - - FUNC_ENTER_NOAPI(FAIL) - - HDassert(/*parameter check*/); - - if (H5X_other_call() < 0) - HGOTO_ERROR(H5E_MAJ, H5E_MIN, FAIL, "badness"); - -done: - if (ret_value < 0) - /* do error cleanup */ - /* do regular cleanup stuff */ - - FUNC_LEAVE_NOAPI(ret_value); -} -``` - -There are a couple of things to note here. - -* Each function call has a header comment block. The information you'll find in - most function comments is not particularly helpful. We're going to improve the - format of this. -* Most functions will return `herr_t` or `hid_t` ID. We try to avoid other - return types and instead use out parameters to return things to the user. -* The name will be of the form `H5X_do_something()` with one or two underscores - after the `H5X`. The naming scheme will be explained later. -* Even though C99 allows declarations anywhere in the function, we put most of - them at the top of the file, with the exception of loop variables and - variables that are "in scope" inside an ifdef. -* We generally initialize values that may need to be freed or otherwise cleaned - up to a "bad" value like `NULL` or `H5I_INVALID_HID` so we can better clean up - resources on function exit, especially when there have been errors. -* Most non-void functions will declare a variable called `ret_value`. This is - used by the error handling macros. It's usually the last thing declared in - the variable block. -* Every function starts with a `FUNC_ENTER macro`, discussed later in this - document. -* Most internal calls will check parameters via asserts. -* We check the return value of any call that can return an error, using the form - shown. -* On errors, an error macro is invoked. These are described later in this - document. -* Any function that returns an error will have a `done` target. Most error - macros jump to this location on errors. -* We do most cleanup at the end of the function, after the `done` target. There - are special `DONE` flavors of error macro that we use in post-`done` cleanup - code to detect and report errors without loops. -* Almost every function ends with a `FUNC_LEAVE` macro. - -And here's an example of a public API call: - -```c -/* - * Doxygen stuff goes here - */ -herr_t -H5Xdo_api_stuff(/*parameters*/) -{ - /* variables go here */ - herr_t ret_value = SUCCEED; - - FUNC_ENTER_API(FAIL) - H5TRACE3(/*stuff*/) - - if (/*parameter check*/) - HGOTO_ERROR(H5E_ARGS, H5E_BADVALUE, FAIL, "badness"); - - /* VOL setup */ - - if (H5VL_call() < 0) - HGOTO_ERROR(H5E_FOO, H5E_BAR, FAIL, "badness"); - -done: - if (ret_value < 0) - /* do error cleanup */ - /* do regular cleanup stuff */ - - FUNC_LEAVE_API(ret_value); -} - -``` - -A public API call differs little from an internal call. The biggest differences: - -* Public API calls are commented using Doxygen. This is how we generate the - reference manual entries. -* The name will have the form `H5Xdo_stuff()`, with no underscores after the `H5X`. -* The function enter macro is `FUNC_ENTER_API` (or similar). Under the hood, this - one differs quite a bit from an internal function enter macro. It checks for - package initialization, for example, and acquires the global lock in thread-safe HDF5. -* There is a `TRACE` macro. This helps with API tracing and is applied by a - script invoked by `autogen.sh` (Autotools) or CMake. You probably don't need - to worry much about this. -* Parameter checking uses the regular HDF5 error scheme and invokes - `HGOTO_ERROR` macros on errors. -* In storage-related calls, there will usually be some VOL setup (HDF5 1.12.x - and later) and in lieu of a regular internal API call, there will be an `H5VL` - VOL call. -* The function exit macro will be `FUNC_LEAVE_API` (or similar). This is where - we release the global thread-safe lock, etc. - - -### Public, private, package - -HDF5 is divided into _packages_, which encapsulate related functionality. Each -has a prefix of the form `H5X(Y)`. An example is the dataset package, which has -the prefix `H5D`. Hopefully, you are familiar with this from the public API. In -addition to the public packages, we all know and love, there are many internal -packages that are not visible to the public via API calls, like `H5FL` (free -lists / memory pools) and `H5B2` (version 2 B-trees). There's also an `H5` -package that deals with very low-level things like library startup. - -API calls, types, etc. in HDF5 have three levels of visibility. From most to -least visible, these are: - -* Public -* Private -* Package - -**Public** things are in the public API. They are usually found in `H5Xpublic.h` -header files. API calls are of the form `H5Xfoo()`, with no underscores between -the package name and the rest of the function name. - -**Private** things are for use across the HDF5 library, and can be used outside the packages -that contain them. They collectively make up the "internal library API". API -calls are of the form `H5X_foo()` with one underscore between the package -name and the rest of the function name. - -**Package** things are for use inside the package and the compiler will -complain if you include them outside of the package they belong to. They -collectively make up the "internal package API". API calls are of the form -`H5X__foo()` with *two* underscores between the package name and the rest of the -function name. The concept of "friend" packages exists and you can declare this -by defining `_FRIEND` in a file. This will let you include the package -header from a package in a file that it is not a member of. Doing this is -strongly discouraged, though. Test functions are often declared in package -headers as they expose package internals and test programs can include -multiple package headers so they can check on internal package state. - -Note that the underscore scheme is primarily for API calls and does not extend -to things like types and symbols. Another thing to keep in mind is that the -difference between package and private API calls can be somewhat arbitrary. -We're hoping to improve the coherence of the internal APIs via refactoring. - - -### Function enter and leave macros - -Function enter and leave macros are added to almost all HDF5 API calls. This is -where we set up error handling (see below) and things like the thread-safety -global lock (in public API calls). There are different flavors depending on the -API call and it's very important that they are appropriate for the function they -mark up. - -The various combinations you are most likely to encounter: - -|Macro|Use| -|-----|---| -|`FUNC_ENTER_API`|Used when entering a public API call| -|`FUNC_ENTER_NOAPI`|Used when entering a private API call| -|`FUNC_ENTER_PACKAGE`|Used when entering a package API call| - -There are also `_NO_INIT` flavors of some of these macros. These are usually -small utility functions that don't initialize the library, like -`H5is_library_threadsafe()`. They are uncommon. - -You may also come across `_NO_FS` ("no function stack") flavors that don't push -themselves on the stack. These are rare. - -For the most part, you will be using the `API`, `NOAPI`, and `PACKAGE` macros. - -You may see other versions if you are working in a maintenance branch, like the -`STATIC` macros that we removed in 1.13. We've been working to reduce the -complexity and number of these macros and we don't always push that downstream -due to the scope of the changes involved. You should be able to figure out what -any new macros do based on what you've seen here, though. - -### Error macros - -Almost all HDF5 functions return an error code, usually -1 or some typedef'd -equivalent. Functions that return `void` should be avoided, even if the function -cannot fail. Instead, return an `herr_t` value and always return `SUCCEED`. - -|Type|Error Value| -|----|-----------| -|`herr_t`|`FAIL`| -|any signed integer type|-1| -|`hid_t`|`H5I_INVALID_HID`| -|`htri_t`|`FAIL`| -|`haddr_t`|`HADDR_UNDEF`| -|pointer|`NULL`| - -We've been trying to move away from using anything other than `herr_t` or `hid_t` -to return errors, as eliminating half of a variable's potential values just so -we can return a 'bad' value on errors seems unwise in a library that is -designed to scale. - -`herr_t` is a typedef'd signed integer. In the library, we only define two -values for it: `SUCCEED` and `FAIL`, which are defined to 0 and -1, respectively, -in `H5private.h`. We do not export these values, so public API calls just note -that `herr_t` values will be negative on failure and non-negative on success. - -Most of the error handling is performed using macros. The only extra setup you -will have to do is: - -1. Create a variable named `ret_value` with the same type -as the return value for the function. If the type is `herr_t` it is frequently -set to `SUCCEED` and will be set to `FAIL` on errors. In most other cases, -the value is initialized to the 'bad' value and the function's code will set -`ret_value` to a 'good' value at some point, with errors setting it back to -the 'bad' value. - -2. Create a done target (`done:`) just before you start your error cleanup. -This will be the point to which the error macros will jump. - -We check for errors on almost all internal lines of C code that could putatively -fail. The general format is this: - -```c -if (function_that_could_fail(foo, bar) < 0) - HGOTO_ERROR(H5E_, H5E_, , "tell me about badness"); -``` - -`HGOTO_ERROR` is one of a set of macros defined in `H5Eprivate.h`. This macro -pops an error on the error stack and sets the return value to ``, -then jumps to the `done:` target. - -Major and minor codes are a frequent cause of confusion. A full list of them -can be found in `H5err.txt`, which is processed into the actual error header -files at configure time by the `bin/make_err` script. The original intent was for major and minor error -codes to be strongly associated. i.e., a given minor code would *only* be used -with its associated major code. Unfortunately, this has not been the case in -practice, and the emitted text can appear nonsensical in error -stack dumps. Even worse, the major and minor error codes are used inconsistently -throughout the library, making interpreting them almost impossible for -external users. We hope to address this deficiency in the near future. - -In the meantime, the following guidelines can be helpful: - -1. Use `H5E_ARGS` as the major error category when parsing function parameters. The minor code will usually be `H5E_BADVALUE`, `H5E_BADRANGE`, or `H5E_BADTYPE`. -2. Otherwise use the same major code throughout the source file. There is almost a 1-1 correspondence between packages and major error codes. -3. Pick the minor code that seems to match the API call. You can grep through the library to find similar uses. -4. The string at the end of the `HGOTO_ERROR` macro is much more important, so make sure that is helpful - -You will still sometimes see the major error code match the package of a failing -function call. We're trying to fix those as we come across them. - -Since the `HGOTO_ERROR` macro jumps to the `done:` target, you can't use it -after the `done:` target without creating a loop. Instead, you'll need to use -the `HDONE_ERROR` macro, which will handle errors without jumping to the target. -Instead, processing will continue after pushing the error and setting the -return value, in the hopes that we can clean up as much as possible. - -At the end of the function, the `FUNC_LEAVE` macro will return `ret_value`. - -### Trace macros - -These are automatically generated for public C library API calls by the -`bin/trace` script, which scans the source code, looking for functions of the -form `H5X(Y?)()`, to which it will add or update the `H5TRACE` macros. - -`H5TRACE` macros are only added to public C library API calls. They are NOT -added to the language wrappers, tools code, high-level library, etc. - -You should never have to modify an `H5TRACE` macro. Either point `bin/trace` at -your source file or run `autogen.sh` (which runs `bin/trace` over the C files -in `src`). `bin/trace` is a Perl script, so you'll need to have that available. - - -### Memory - `H5MM` and `H5FL` - -In the C library itself, we use `H5MM` and `H5FL` calls to allocate and free -memory instead of directly using the standard C library calls. - -The `H5MM` package was originally designed so that it would be easy to swap in a -memory allocator of the developer's choosing. In practice, this has rarely -been a useful feature, and we are thinking about removing this scheme. In -the meantime, almost all memory allocations in the C library will use the -`H5MM` (or `H5FL`) package. - -In the past, we added memory allocation sanity checking to the `H5MM` calls -which added heap canaries to memory allocations and performed sanity checking -and gathered statistics. These were turned on by default in debug builds -for many years. Unfortunately, there is interplay between library-allocated -and externally-allocated memory in the filter pipeline where the heap canaries -can easily get corrupted and cause problems. We also have some API calls that -return library-allocated memory to the user, which can cause problems if they -then use `free(3)` to free it. Given these problems, we now have the sanity -checks turned off by default in all build modes. You can turn them back on via -configure/CMake options, but it's normally easier to use external tools like -valgrind or the compiler's memory debugging options. - -`H5FL` provides memory pools (*Free Lists*) that create a set of fixed-size allocations -of a certain type that the library will reuse as needed. They use `H5MM` calls -under the hood and can be useful when the library creates and frees a lot of -objects of that type. It's difficult to give a good guideline as to when to use -the `H5FL` calls and when to use the `H5MM` calls, but it's probably best to -lean towards `H5MM` unless you can identify a clear performance hit due to -memory cycling. Current library usage can be a good guide, but keep in mind that -the free lists are probably overused in the library. Another thing to keep in -mind is that the free lists can hide memory errors, like use-after-free. Some -developers always turn them off and you'll need to turn them off when running -memory checkers like valgrind. - -Using free list calls differs little from using `H5MM` calls. There are -equivalents for `malloc(3)`, `calloc(3)`, and `free(3)`: - -| C Library|`H5FL`| -|----------|------| -|`malloc`|`H5FL_MALLOC`| -|`calloc`|`H5FL_CALLOC`| -|`free`|`H5FL_FREE`| - -Since free lists provide pre-allocated memory of a fixed size, you can't -reallocate free list memory and there's no `H5FL` `realloc(3)` equivalent. - -You'll also need to add a setup macro to the top of the file. There are a few -flavors defined in `H5FLprivate.h`. Each creates global free list variables, -so there are flavors for extern, static, etc. - -|Macro|Purpose| -|-----|-------| -|`H5FL_DEFINE`|Define a free list that will be used in several files| -|`H5FL_EXTERN`|Define a free list that was defined in another file| -|`H5FL_DEFINE_STATIC`|Define a free list that will only be used in this file| - -You will also see `ARR`, `BLK`, `SEQ`, and `FAC` flavors of the macros. Their -use is beyond the scope of a guide for beginners. - -## Testing - -### Two macro schemes - -The HDF5 C library is tested via a collection of small programs in the `test/` -directory. There are a few schemes in use: - -- `testhdf5` - A larger, composite test program composed of several test files, most of which start with 't' (e.g., `tcoords.c`) -- Shell/Powershell scripts that test things like SWMR and flush/refresh behavior. These scripts run small sub-programs. -- Everything else. These are self-contained test programs that are built and run independently by the test harness. - - -The test programs do not use a standard test framework like cppunit, but instead -use HDF5-specific macros to set up the tests and report errors. There are two -sets of macros, one in `testhdf5.h` and another in `h5test.h`. -Originally, the `testhdf5` programs used the macros in `testhdf5.h` and everything -else used the macros in `h5test.h`, but over time the tests have evolved so that -all tests usually include both headers. - -This is unfortunate, because it's very important to not mix up the "test framework -macros" in each scheme. The `testhdf5.h` macros register errors by setting global -variables and normally continue with the test when they encounter errors. The -`h5test.h` macros indicate errors by jumping to an error target and returning -a `FAIL` `herr_t` value. If you combine these two macro sets, you may -accidentally create tests that fail but do not register the failure. - -We are aware that our testing scheme needs some work and we'll be working to -improve it over the next year or so. - -The command-line tools are tested using a different scheme and are discussed elsewhere. - -### `testhdf5.h` - -The macros in this file are almost exclusively used in the `testhdf5` program. -They register errors by incrementing a global error count that is inspected at -the end of each test (not each test *function*, but each *test* - e.g., after -`test_file()` in `tfile.c` runs, but not when the individual functions it -calls run). - -Test functions generally look like this: - -```c -static void -test_something() -{ - /* Variables go here */ - int out = -1; - herr_t ret; - - MESSAGE(5, ("Testing a thing\n")); - - ret = H5Xsome_api_call(&out); - CHECK(ret, FAIL, "H5Xsome_api_call()"); - VERIFY(out, 6, "incorrect value for out"); -} - -``` - -The `MESSAGE` macro just dumps the name of what we're testing when we've the -verbosity cranked up. The confusingly-named `CHECK` and `VERIFY` macros are -members of a suite of check macros in `testhdf5.h`. `CHECK` macros check to -see if a variable is **NOT** a value, `VERIFY` macros check to see if a variable -**IS** a value. There are different flavors of macro to match different types -so be sure to use the right one to avoid compiler warnings and spurious errors. -Under the hood, these macros will emit error information and increment the -global error variables. - -Tests are added to `testhdf5` in `testhdf5.c` using the `AddTest()` call. -Each test will have a driver function, usually named something like `test_()` -that invokes the test functions in the file. -Most tests will cleanup their files using a `cleanup_()` call. If you are -deleting HDF5 files, you should use `H5Fdelete()` instead of `remove(3)` so -that files can be cleaned even when they use alternative VFDs or VOL connectors. -You'll also need to add prototypes for any new test driver or cleanup functions -to `testhdf5.h`. - -Because these macros interact with global variables that are only used in the -testhdf5 program, they are useless anywhere else in the library. Even worse, it -will *look* like you are testing functionality, but errors will not be picked -up by the non-testhdf5 programs, hiding problems. - -### `h5test.h` - -These are the most commonly used macros and are used throughout the test code, -even in places that are not specifically tests. Unlike the scheme used in -the `testhdf5` program, these macros work more like the library, jumping to -an `error:` target on errors. There is no common `ret_value` variable, however. - -Test functions will usually look like this: - -```c -static herr_t -test_something() -{ - hid_t fapl_id = H5I_INVALID_HID; - hid_t fid = H5I_INVALID_HID; - char filename[1024]; - int *buf = NULL; - - TESTING("Testing some feature"); - - if ((fapl_id = h5_fileaccess()) < 0) - TEST_ERROR; - h5_fixname(FILENAME[0], fapl, filename, sizeof(filename)); - - if ((fid = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id)) < 0) - TEST_ERROR; - - /* Many more calls here */ - - PASSED(); - return SUCCEED; - -error: - - HDfree(buf); - - H5E_BEGIN_TRY - { - H5Fclose(fid); - } - H5E_END_TRY - - return FAIL; -} -``` - -Tests begin with a `TESTING` macro that emits some text (unlike the `testhdf5` -case, this is always dumped). Any errors will be handled by one of the -`TEST_ERROR` macros. For the most part, `TEST_ERROR` will suffice, but there -are others in `h5test.h` if you want to emit custom text, dump the HDF5 error -stack when it would not normally be triggered, etc. - -Most tests will be set up to run with arbitrary VFDs. To do this, you set the -fapl ID using the `h5_fileaccess()` function, which will check the `HDF5_TEST_DRIVER` -environment variable and set the fapl's VFD accordingly. The `h5_fixname()` -call can then be used to get a VFD-appropriate filename for the `H5Fcreate()`, -etc. call. - -In the `error` section, we clean up resources and return a 'bad' value, which -will usually either be `FAIL` or -1. - -The `main()` function of each test program will usually start out by calling -`h5_reset()`, then run each test, incrementing an error count variable if a -test fails. The exit code and text will be set based on the error count. - -### Scripts - -If you need to fire off multiple programs to test a new feature, you may have -to do this via a script. These are normally named `test_.sh.in`. The -`.in` is because the scripts are often modified and copied during the configure -step. In the past, we have tried to stick to POSIX Bourne shell scripts, but -many scripts now require bash. - -If you write a new test script, it is important to also add a PowerShell -equivalent for testing on Windows. - -It's helpful to run any new shell scripts through `shellcheck` -(https://www.shellcheck.net/) to ensure that your scripts are free from -common problems. - - -### Parallel tests (in `testpar/`) - -To be covered in a future update of this guide... - -### Adding new tests - -All new HDF5 library functionality (including bugfixes) should have a test. -Some rules of thumb: - -- If you need to run multiple programs, you'll need to create a script and some test programs. Use the macros in `h5test.h` to handle errors in your test programs. -- If a suitable test program already exists (especially if your tests will be small), add your new tests to the existing file. -- If you need to create a new test program, create one that uses the `h5test.h` macros. -- Avoid adding new tests to `testhdf5` or using the macros in `testhdf5.h`. - -Don't forget that you'll need to add your test program or script to the lists in -both the CMake and Autotools test files (`CMakeLists.txt` and `Makefile.am` in -`test/` respectively). For simple tests, you just need to add your new test to -the list of tests. - -All new tests **MUST** run under both the Autotools and CMake. Ideally, they -should also work on Windows, but we're willing to overlook this for things -that are unlikely to be useful on that platform. - -## Documentation - -We have moved the user guide and reference manual to Doxygen. All public API -calls and symbols should have Doxygen markup in the public header file. New major -features should be documented in the user guide. This Doxygen content is located in -the package's module header file (`H5Xmodule.h`). Language wrapper calls -(C++/Fortran/Java) should also have Doxygen markup, which will be located with -the wrapper source code. Images and other common Doxygen files belong in the -`doxygen` directory. - -Internal documentation for developer consumption is currently stored as Markdown -files in the `doc` directory. This may change in the future. Documentation that -helps understand the contents of a directory is often stored in a README.md -Markdown file in that directory. - -Build and install documentation is stored in text files in `release_docs`. This -is admittedly not the best place for this. History files are also kept here. - -## Command-Line Tools - -The HDF5 command-line tools are written in C and built with the library by default. -The code is organized into a central tools library (in the `tools/lib` directory) -that includes some common functionality and the individual programs, each of which -has its own directory in `tools/src`. A few of the smaller tools are aggregated -into the `tools/src/misc` directory. Only h5diff has a parallel version at this -time and the parallel-specific functionality is in the `tools/src/h5diff/ph5diff_main.c` file. -Some h5diff functionality has also made its way into the tools library. -The tools code is not as well organized as the main library, so there's more opportunity -for refactoring here. - -Also worth noting is that the command-line tools only use **public** HDF5 API -calls, even though they include `H5private.h` in order to take advantage of -the platform-independence scheme we use in the main library and some private -utility functions. - -There are also a few tools in the high-level library. The gif2h5 and h52gif tools -are poorly-written and have known security issues, so they will soon be moved -to a separate repository, leaving h5watch as the only high-level command-line tool. - -### Source code - -The source code for the tools likes more like standard C code and uses its own -set of macros, which are defined in the tools library header files. There are -no `FUNC_ENTER` macros, you do not need to define a `ret_value` variable, -and the error macros are greatly simplified. Errors are usually handled by -a `TOOLS_ERROR` macro (or `TOOLS_GOTO_ERROR` if you need to jump to a `done:` -target to handle cleanup). -One area where the tools need a lot more work is in handling errors. The tools -code frequently ignores errors, often in functions that return `void`. - -A "tools-only" consideration is the use of command-line arguments. We try to -be conservative about these, even though they really aren't in the "public API" -in the same way as API calls are. Additions and changes to the options will -probably result in some discussion. - -### Tools tests - -In most cases, a tool will be run against an input HDF5 file with a particular -set of command-line parameters, the exit code checked, and the output compared -with a standard output file. In some cases, errors are expected and standard -error files will be compared. These standard error files often contain HDF5 -error stack dumps, which can cause spurious tool test "failures" when we -make changes to the main HDF5 C library. - -Test files can be located in a few places in the `tools` directory tree. -Common input files that are used with multiple tools are kept in `tools/testfiles`. -Input files that are used with just one tool are located in `tools/test//testfiles`. -Expected output files are located with their respective HDF5 files and end in `.txt`. -Expected error files are also located with their respective HDF5 files and end in `.err`. -h5dump files will usually have `.ddl` and `.xml` output files and there will -usually be `.` files that contain help output. -The test files are generated by programs with `gentest` in their name, though -we typically check the generated HDF5 files in instead of recreating them -as a part of running the tools tests. - -The Autotools aggregate the tools tests in per-tool shell scripts in the -`tools/test/` directory. Each script starts with a few utility functions -that perform setup, compare files, clean output, etc. and the test commands -will appear at the end. CMake works similarly, but each test is set up in -the CMakeLists.txt and CMakeTests.cmake files in the same directory. - -Adding a new test will usually involve: -- Adding a new function to the appropriate generator program to create a new HDF5 file -- Adding your new test to the CMake and Autotools test scripts, as described above -- Adding appropriate output and/or error files for comparison - -You MUST add new tests to both the Autotools and CMake. diff --git a/doc/parallel-compression.md b/doc/parallel-compression.md index dc68ea4e081..ae05b2090bd 100644 --- a/doc/parallel-compression.md +++ b/doc/parallel-compression.md @@ -79,7 +79,7 @@ participate in the collective write call. ## Multi-dataset I/O support The parallel compression feature is supported when using the -multi-dataset I/O API routines ([H5Dwrite_multi](https://hdfgroup.github.io/hdf5/group___h5_d.html#gaf6213bf3a876c1741810037ff2bb85d8)/[H5Dread_multi](https://hdfgroup.github.io/hdf5/group___h5_d.html#ga8eb1c838aff79a17de385d0707709915)), but the +multi-dataset I/O API routines ([H5Dwrite_multi](https://hdfgroup.github.io/hdf5/v1_14/group___h5_d.html#gaf6213bf3a876c1741810037ff2bb85d8)/[H5Dread_multi](https://hdfgroup.github.io/hdf5/v1_14/group___h5_d.html#ga8eb1c838aff79a17de385d0707709915)), but the following should be kept in mind: - Parallel writes to filtered datasets **must** still be collective, @@ -99,7 +99,7 @@ following should be kept in mind: ## Incremental file space allocation support -HDF5's [file space allocation time](https://hdfgroup.github.io/hdf5/group___d_c_p_l.html#ga85faefca58387bba409b65c470d7d851) +HDF5's [file space allocation time](https://hdfgroup.github.io/hdf5/v1_14/group___d_c_p_l.html#ga85faefca58387bba409b65c470d7d851) is a dataset creation property that can have significant effects on application performance, especially if the application uses parallel HDF5. In a serial HDF5 application, the default file space @@ -118,7 +118,7 @@ While this strategy has worked in the past, it has some noticeable drawbacks. For one, the larger the chunked dataset being created, the more noticeable overhead there will be during dataset creation as all of the data chunks are being allocated in the HDF5 file. -Further, these data chunks will, by default, be [filled](https://hdfgroup.github.io/hdf5/group___d_c_p_l.html#ga4335bb45b35386daa837b4ff1b9cd4a4) +Further, these data chunks will, by default, be [filled](https://hdfgroup.github.io/hdf5/v1_14/group___d_c_p_l.html#ga4335bb45b35386daa837b4ff1b9cd4a4) with HDF5's default fill data value, leading to extraordinary dataset creation overhead and resulting in pre-filling large portions of a dataset that the application might have been planning @@ -126,7 +126,7 @@ to overwrite anyway. Even worse, there will be more initial overhead from compressing that fill data before writing it out, only to have it read back in, unfiltered and modified the first time a chunk is written to. In the past, it was typically suggested that parallel -HDF5 applications should use [H5Pset_fill_time](https://hdfgroup.github.io/hdf5/group___d_c_p_l.html#ga6bd822266b31f86551a9a1d79601b6a2) +HDF5 applications should use [H5Pset_fill_time](https://hdfgroup.github.io/hdf5/v1_14/group___d_c_p_l.html#ga6bd822266b31f86551a9a1d79601b6a2) with a value of `H5D_FILL_TIME_NEVER` in order to disable writing of the fill value to dataset chunks, but this isn't ideal if the application actually wishes to make use of fill values. @@ -220,14 +220,14 @@ chunks to end up at addresses in the file that do not align well with the underlying file system, possibly leading to poor performance. As an example, Lustre performance is generally good when writes are aligned with the chosen stripe size. -The HDF5 application can use [H5Pset_alignment](https://hdfgroup.github.io/hdf5/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a) +The HDF5 application can use [H5Pset_alignment](https://hdfgroup.github.io/hdf5/v1_14/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a) to have a bit more control over where objects in the HDF5 file end up. However, do note that setting the alignment of objects generally wastes space in the file and has the potential to dramatically increase its resulting size, so caution should be used when choosing the alignment parameters. -[H5Pset_alignment](https://hdfgroup.github.io/hdf5/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a) +[H5Pset_alignment](https://hdfgroup.github.io/hdf5/v1_14/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a) has two parameters that control the alignment of objects in the HDF5 file, the "threshold" value and the alignment value. The threshold value specifies that any object greater @@ -264,19 +264,19 @@ in a file, this can create significant amounts of free space in the file over its lifetime and eventually cause performance issues. -An HDF5 application can use [H5Pset_file_space_strategy](https://hdfgroup.github.io/hdf5/group___f_c_p_l.html#ga167ff65f392ca3b7f1933b1cee1b9f70) +An HDF5 application can use [H5Pset_file_space_strategy](https://hdfgroup.github.io/hdf5/v1_14/group___f_c_p_l.html#ga167ff65f392ca3b7f1933b1cee1b9f70) with a value of `H5F_FSPACE_STRATEGY_PAGE` to enable the paged aggregation feature, which can accumulate metadata and raw data for dataset data chunks into well-aligned, configurably sized "pages" for better performance. However, note that using the paged aggregation feature will cause any setting from -[H5Pset_alignment](https://hdfgroup.github.io/hdf5/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a) +[H5Pset_alignment](https://hdfgroup.github.io/hdf5/v1_14/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a) to be ignored. While an application should be able to get -comparable performance effects by [setting the size of these pages](https://hdfgroup.github.io/hdf5/group___f_c_p_l.html#gad012d7f3c2f1e1999eb1770aae3a4963) to be equal to the value that -would have been set for [H5Pset_alignment](https://hdfgroup.github.io/hdf5/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a), +comparable performance effects by [setting the size of these pages](https://hdfgroup.github.io/hdf5/v1_14/group___f_c_p_l.html#gad012d7f3c2f1e1999eb1770aae3a4963) to be equal to the value that +would have been set for [H5Pset_alignment](https://hdfgroup.github.io/hdf5/v1_14/group___f_a_p_l.html#gab99d5af749aeb3896fd9e3ceb273677a), this may not necessarily be the case and should be studied. -Note that [H5Pset_file_space_strategy](https://hdfgroup.github.io/hdf5/group___f_c_p_l.html#ga167ff65f392ca3b7f1933b1cee1b9f70) +Note that [H5Pset_file_space_strategy](https://hdfgroup.github.io/hdf5/v1_14/group___f_c_p_l.html#ga167ff65f392ca3b7f1933b1cee1b9f70) has a `persist` parameter. This determines whether or not the file free space manager should include extra metadata in the HDF5 file about free space sections in the file. If this @@ -300,12 +300,12 @@ hid_t file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, fcpl_id, fapl_id); While the parallel compression feature requires that the HDF5 application set and maintain collective I/O at the application -interface level (via [H5Pset_dxpl_mpio](https://hdfgroup.github.io/hdf5/group___d_x_p_l.html#ga001a22b64f60b815abf5de8b4776f09e)), +interface level (via [H5Pset_dxpl_mpio](https://hdfgroup.github.io/hdf5/v1_14/group___d_x_p_l.html#ga001a22b64f60b815abf5de8b4776f09e)), it does not require that the actual MPI I/O that occurs at the lowest layers of HDF5 be collective; independent I/O may perform better depending on the application I/O patterns and parallel file system performance, among other factors. The -application may use [H5Pset_dxpl_mpio_collective_opt](https://hdfgroup.github.io/hdf5/group___d_x_p_l.html#gacb30d14d1791ec7ff9ee73aa148a51a3) +application may use [H5Pset_dxpl_mpio_collective_opt](https://hdfgroup.github.io/hdf5/v1_14/group___d_x_p_l.html#gacb30d14d1791ec7ff9ee73aa148a51a3) to control this setting and see which I/O method provides the best performance. @@ -318,7 +318,7 @@ H5Dwrite(..., dxpl_id, ...); ### Runtime HDF5 Library version -An HDF5 application can use the [H5Pset_libver_bounds](https://hdfgroup.github.io/hdf5/group___f_a_p_l.html#gacbe1724e7f70cd17ed687417a1d2a910) +An HDF5 application can use the [H5Pset_libver_bounds](https://hdfgroup.github.io/hdf5/v1_14/group___f_a_p_l.html#gacbe1724e7f70cd17ed687417a1d2a910) routine to set the upper and lower bounds on library versions to use when creating HDF5 objects. For parallel compression specifically, setting the library version to the latest available diff --git a/doxygen/Doxyfile.in b/doxygen/Doxyfile.in index 7c277241cd8..29033f5de21 100644 --- a/doxygen/Doxyfile.in +++ b/doxygen/Doxyfile.in @@ -710,6 +710,7 @@ RECURSIVE = YES # run. EXCLUDE = + # The EXCLUDE_SYMLINKS tag can be used to select whether or not files or # directories that are symbolic links (a Unix file system feature) are excluded # from the input. @@ -736,7 +737,6 @@ EXCLUDE_PATTERNS += */hl/fortran/src/*.h EXCLUDE_PATTERNS += */HDF5Examples/FORTRAN/* EXCLUDE_PATTERNS += */sanitizer/* - # The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names # (namespaces, classes, functions, etc.) that should be excluded from the # output. The symbol name can be a fully qualified name, a word, or if the diff --git a/doxygen/aliases b/doxygen/aliases index e2191cd4bd5..84c20d5ba24 100644 --- a/doxygen/aliases +++ b/doxygen/aliases @@ -1,5 +1,20 @@ ALIASES += THG="The HDF Group" +################################################################################ +# Default URLs (Note that md files do not use any aliases) +################################################################################ +ALIASES += HDFURL="docs.hdfgroup.org/hdf5" +ALIASES += ARCURL="docs.hdfgroup.org/archive/support/HDF5/doc" +ALIASES += RFCURL="docs.hdfgroup.org/hdf5/rfc" +ALIASES += DSPURL="portal.hdfgroup.org/display/HDF5" +ALIASES += DOCURL="portal.hdfgroup.org/documentation/hdf5-docs" +ALIASES += DWNURL="portal.hdfgroup.org/downloads" +ALIASES += AEXURL="support.hdfgroup.org/ftp/HDF5/examples" +# doxygen subdir (develop, v1_14) +ALIASES += DOXURL="hdfgroup.github.io/hdf5/v1_14" +#branch name (develop, hdf5_1_14) +ALIASES += SRCURL="github.com/HDFGroup/hdf5/blob/hdf5_1_14" + ################################################################################ # Styling ################################################################################ @@ -234,16 +249,16 @@ ALIASES += sa_metadata_ops="\sa \li H5Pget_all_coll_metadata_ops() \li H5Pget_co # References ################################################################################ -ALIASES += ref_cons_semantics="Enabling a Strict Consistency Semantics Model in Parallel HDF5" -ALIASES += ref_file_image_ops="HDF5 File Image Operations" -ALIASES += ref_filter_pipe="Data Flow Pipeline for H5Dread()" -ALIASES += ref_group_impls="Group implementations in HDF5" -ALIASES += ref_h5lib_relver="HDF5 Library Release Version Numbers" -ALIASES += ref_mdc_in_hdf5="Metadata Caching in HDF5" -ALIASES += ref_mdc_logging="Metadata Cache Logging" -ALIASES += ref_news_112="New Features in HDF5 Release 1.12" -ALIASES += ref_h5ocopy="Copying Committed Datatypes with H5Ocopy()" -ALIASES += ref_sencode_fmt_change="RFC H5Sencode() / H5Sdecode() Format Change" +ALIASES += ref_cons_semantics="Enabling a Strict Consistency Semantics Model in Parallel HDF5" +ALIASES += ref_file_image_ops="HDF5 File Image Operations" +ALIASES += ref_filter_pipe="Data Flow Pipeline for H5Dread()" +ALIASES += ref_group_impls="Group implementations in HDF5" +ALIASES += ref_h5lib_relver="HDF5 Library Release Version Numbers" +ALIASES += ref_mdc_in_hdf5="Metadata Caching in HDF5" +ALIASES += ref_mdc_logging="Metadata Cache Logging" +ALIASES += ref_news_112="New Features in HDF5 Release 1.12" +ALIASES += ref_h5ocopy="Copying Committed Datatypes with H5Ocopy()" +ALIASES += ref_sencode_fmt_change="RFC H5Sencode() / H5Sdecode() Format Change" ALIASES += ref_vlen_strings="\Emph{Creating variable-length string datatypes}" ALIASES += ref_vol_doc="VOL documentation" @@ -251,103 +266,103 @@ ALIASES += ref_vol_doc="VOL documentation" # RFCs ################################################################################ -ALIASES += ref_rfc20220819="Terminal VOL Connector Feature Flags" -ALIASES += ref_rfc20210528="Multi-Thread HDF5" -ALIASES += ref_rfc20210219="Selection I/O" -ALIASES += ref_rfc20200213="VFD Sub-filing" -ALIASES += ref_rfc20200210="Onion VFD" -ALIASES += ref_rfc20190923="Virtual Object Layer (VOL) API Compatibility" -ALIASES += ref_rfc20190715="Variable-Length Data in HDF5 Sketch Design" -ALIASES += ref_rfc20190410="A Plugin Interface for HDF5 Virtual File Drivers" -ALIASES += ref_rfc20181231="Dataset Object Header Size" -ALIASES += ref_rfc20181220="MS 3.2 – Addressing Scalability: Scalability of open, close, flush CASE STUDY: CGNS Hotspot analysis of CGNS cgp_open" -ALIASES += ref_rfc20180830="Sparse Chunks" -ALIASES += ref_rfc20180829="H5FD_MIRROR Virtual File Driver" -ALIASES += ref_rfc20180815="Splitter_VFD" -ALIASES += ref_rfc20180712="Update to HDF5 References" -ALIASES += ref_rfc20180620="Chunk query functionality in HDF5" -ALIASES += ref_rfc20180610="VFD SWMR" -ALIASES += ref_rfc20180321="API Contexts" -ALIASES += ref_rfc20180125="Enhancement to the tool h5clear" -ALIASES += ref_rfc20170707="H5Sencode/H5Sdecode Format Change" -ALIASES += ref_rfc20160105="Setting Bounds for Object Creation in HDF5 1.10.0" -ALIASES += ref_rfc20150915="File Format Changes in HDF5 1.10.0" -ALIASES += ref_rfc20150709="Page Buffering" -ALIASES += ref_rfc20150615="Metadata Cache Image" -ALIASES += ref_rfc20150429="New Datatypes" -ALIASES += ref_rfc20150424="Collective Metadata Writes" -ALIASES += ref_rfc20150423="Enabling Collective Metadata Reads" -ALIASES += ref_rfc20150301="The Tool to Handle HDF5 File Format Compatibility for Chunked Datasets" -ALIASES += ref_rfc20150212="H5LTget_hardlinks – High-level API to list all the hard links to an object" -ALIASES += ref_rfc20150205="HDF5 Fortran Wrappers Maintenance: Dropping Support for Non-Fortran 2003 Standard Compliant Compilers" -ALIASES += ref_rfc20150202="New Autotools Behavior" -ALIASES += ref_rfc20141210="HDF5 Virtual Dataset" -ALIASES += ref_rfc20141201="Allocate/Free Mismatches in HDF5 Filter Code on Windows" -ALIASES += ref_rfc20140916="Virtual Object Layer" -ALIASES += ref_rfc20140827="Chunking and Compression Performance Tool Requirements" -ALIASES += ref_rfc20140729="Replacing H5Fis_hdf5() with H5Fis_accessible()" -ALIASES += ref_rfc20140722="Switching to a 64-bit hid_t Space in HDF5" -ALIASES += ref_rfc20140717="Data Analysis Extensions" -ALIASES += ref_rfc20140707="Virtual Object Layer" -ALIASES += ref_rfc20140524="HDF5 Compression Demystified" -ALIASES += ref_rfc20140318="Freeing Memory Allocated by the HDF5 Library" -ALIASES += ref_rfc20140313="Options to handle compatibility issues for HDF5 files" -ALIASES += ref_rfc20140224="Design: Metadata Cache Logging" -ALIASES += ref_rfc20131211="Fine-Grained Control of Metadata Cache Flushes" -ALIASES += ref_rfc20130930="Read Attempts for Metadata with Checksum" -ALIASES += ref_rfc20130919="Core VFD Backing Store Paged Writes" -ALIASES += ref_rfc20130630="Flush Dependency Testing" -ALIASES += ref_rfc20130316="HDF5 Dynamically Loaded Filters" -ALIASES += ref_rfc20121114="Direct Chunk Write" -ALIASES += ref_rfc20121024="HDF5 File Space Management" -ALIASES += ref_rfc20120828="New HDF5 API Routines for HPC Applications - Read/Write Multiple Datasets in an HDF5 file" -ALIASES += ref_rfc20120523="HDF5 File Space Management: Paged Aggregation" -ALIASES += ref_rfc20120501="HDF5 File Image Operations" -ALIASES += ref_rfc20120305="Enabling a Strict Consistency Semantics Model in Parallel HDF5" -ALIASES += ref_rfc20120220="h5repack: Improved Hyperslab selections for Large Chunked Datasets" -ALIASES += ref_rfc20120120="A Maintainer's Guide for the Datatype Module in HDF5 Library" -ALIASES += ref_rfc20120104="Actual I/O Mode" -ALIASES += ref_rfc20111119="New public functions to handle comparison" -ALIASES += ref_rfc20110825="Merging Named Datatypes in H5Ocopy()" -ALIASES += ref_rfc20110811="Expanding the HDF5 Hyperslab Selection Interface" -ALIASES += ref_rfc20110726="HDF5 File Space Allocation and Aggregation" -ALIASES += ref_rfc20110614=" Refactor h5dump to Improve Maintenance" -ALIASES += ref_rfc20110329="Support External Link Open File Cache in HDF5 Tools" -ALIASES += ref_rfc20110118="h5diff Attribute Comparisons" -ALIASES += ref_rfc20101122="SWMR Timeouts" -ALIASES += ref_rfc20101104="Caching Files Opened Through External Links" -ALIASES += ref_rfc20101018="HDF5 File and Object Comparison Specification" -ALIASES += ref_rfc20100902="h5edit – An HDF5 File Editing Tool" -ALIASES += ref_rfc20100727="Reserved Characters for HDF5 Applications" -ALIASES += ref_rfc20100726="High-Level HDF5 API routines for HPC Applications" -ALIASES += ref_rfc20100511="h5diff – Exclude Object(s) from Comparison" -ALIASES += ref_rfc20100422="Generating attributes into an object with a tool" -ALIASES += ref_rfc20100312="Supporting HDF5 1.8 in HDF5 Command Line Tools" -ALIASES += ref_rfc20091218="Supporting soft-link and external-link for h5diff" -ALIASES += ref_rfc20090907="HDF5 Tools Library Functions" -ALIASES += ref_rfc20090612="Default EPSILON values for comparing floating point data" -ALIASES += ref_rfc20081218="Reporting of Non-Comparable Datasets by h5diff" -ALIASES += ref_rfc20081205="External Link Traversal Callback" -ALIASES += ref_rfc20081030="Setting Raw Data Chunk Cache Parameters in HDF5" -ALIASES += ref_rfc20080915="Performance Report for Free-space Manager" -ALIASES += ref_rfc20080904="Setting File Access Property List for accessing External Link" -ALIASES += ref_rfc20080728="Native Time Types in HDF5" -ALIASES += ref_rfc20080723="Special Values in HDF5" -ALIASES += ref_rfc20080301="Dynamic Transformations to HDF5 Data" -ALIASES += ref_rfc20080209="Using SVN branching to improve software development process at THG" -ALIASES += ref_rfc20080206="Maintaining the HISTORY.txt and RELEASE.txt files in HDF5" -ALIASES += ref_rfc20071111="Addressing HDF5 file corruption issue" -ALIASES += ref_rfc20071018="NaN detection in HDF5" -ALIASES += ref_rfc20070801="Metadata Journaling to Improve Crash Survivability" -ALIASES += ref_rfc20070413="API Compatibility Strategies for HDF5" -ALIASES += ref_rfc20070115="A \"Private\" Heap for HDF5" -ALIASES += ref_rfc20060623="Performance Comparison of Collective I/O and Independent I/O with Derived Datatypes" -ALIASES += ref_rfc20060604="h5stat tool" -ALIASES += ref_rfc20060505="Simple Performance Test on Fletcher32 Filter" -ALIASES += ref_rfc20060410="Requirement Specifications of an HDF5 File Format Validation Tool" -ALIASES += ref_rfc20060317="Proposed changes to the sec2 driver " -ALIASES += ref_rfc20060124="Mapping FITS data to HDF5" -ALIASES += ref_rfc20040811="Conversion Between Text and Datatype" +ALIASES += ref_rfc20220819="Terminal VOL Connector Feature Flags" +ALIASES += ref_rfc20210528="Multi-Thread HDF5" +ALIASES += ref_rfc20210219="Selection I/O" +ALIASES += ref_rfc20200213="VFD Sub-filing" +ALIASES += ref_rfc20200210="Onion VFD" +ALIASES += ref_rfc20190923="Virtual Object Layer (VOL) API Compatibility" +ALIASES += ref_rfc20190715="Variable-Length Data in HDF5 Sketch Design" +ALIASES += ref_rfc20190410="A Plugin Interface for HDF5 Virtual File Drivers" +ALIASES += ref_rfc20181231="Dataset Object Header Size" +ALIASES += ref_rfc20181220="MS 3.2 – Addressing Scalability: Scalability of open, close, flush CASE STUDY: CGNS Hotspot analysis of CGNS cgp_open" +ALIASES += ref_rfc20180830="Sparse Chunks" +ALIASES += ref_rfc20180829="H5FD_MIRROR Virtual File Driver" +ALIASES += ref_rfc20180815="Splitter_VFD" +ALIASES += ref_rfc20180712="Update to HDF5 References" +ALIASES += ref_rfc20180620="Chunk query functionality in HDF5" +ALIASES += ref_rfc20180610="VFD SWMR" +ALIASES += ref_rfc20180321="API Contexts" +ALIASES += ref_rfc20180125="Enhancement to the tool h5clear" +ALIASES += ref_rfc20170707="H5Sencode/H5Sdecode Format Change" +ALIASES += ref_rfc20160105="Setting Bounds for Object Creation in HDF5 1.10.0" +ALIASES += ref_rfc20150915="File Format Changes in HDF5 1.10.0" +ALIASES += ref_rfc20150709="Page Buffering" +ALIASES += ref_rfc20150615="Metadata Cache Image" +ALIASES += ref_rfc20150429="New Datatypes" +ALIASES += ref_rfc20150424="Collective Metadata Writes" +ALIASES += ref_rfc20150423="Enabling Collective Metadata Reads" +ALIASES += ref_rfc20150301="The Tool to Handle HDF5 File Format Compatibility for Chunked Datasets" +ALIASES += ref_rfc20150212="H5LTget_hardlinks – High-level API to list all the hard links to an object" +ALIASES += ref_rfc20150205="HDF5 Fortran Wrappers Maintenance: Dropping Support for Non-Fortran 2003 Standard Compliant Compilers" +ALIASES += ref_rfc20150202="New Autotools Behavior" +ALIASES += ref_rfc20141210="HDF5 Virtual Dataset" +ALIASES += ref_rfc20141201="Allocate/Free Mismatches in HDF5 Filter Code on Windows" +ALIASES += ref_rfc20140916="Virtual Object Layer" +ALIASES += ref_rfc20140827="Chunking and Compression Performance Tool Requirements" +ALIASES += ref_rfc20140729="Replacing H5Fis_hdf5() with H5Fis_accessible()" +ALIASES += ref_rfc20140722="Switching to a 64-bit hid_t Space in HDF5" +ALIASES += ref_rfc20140717="Data Analysis Extensions" +ALIASES += ref_rfc20140707="Virtual Object Layer" +ALIASES += ref_rfc20140524="HDF5 Compression Demystified" +ALIASES += ref_rfc20140318="Freeing Memory Allocated by the HDF5 Library" +ALIASES += ref_rfc20140313="Options to handle compatibility issues for HDF5 files" +ALIASES += ref_rfc20140224="Design: Metadata Cache Logging" +ALIASES += ref_rfc20131211="Fine-Grained Control of Metadata Cache Flushes" +ALIASES += ref_rfc20130930="Read Attempts for Metadata with Checksum" +ALIASES += ref_rfc20130919="Core VFD Backing Store Paged Writes" +ALIASES += ref_rfc20130630="Flush Dependency Testing" +ALIASES += ref_rfc20130316="HDF5 Dynamically Loaded Filters" +ALIASES += ref_rfc20121114="Direct Chunk Write" +ALIASES += ref_rfc20121024="HDF5 File Space Management" +ALIASES += ref_rfc20120828="New HDF5 API Routines for HPC Applications - Read/Write Multiple Datasets in an HDF5 file" +ALIASES += ref_rfc20120523="HDF5 File Space Management: Paged Aggregation" +ALIASES += ref_rfc20120501="HDF5 File Image Operations" +ALIASES += ref_rfc20120305="Enabling a Strict Consistency Semantics Model in Parallel HDF5" +ALIASES += ref_rfc20120220="h5repack: Improved Hyperslab selections for Large Chunked Datasets" +ALIASES += ref_rfc20120120="A Maintainer's Guide for the Datatype Module in HDF5 Library" +ALIASES += ref_rfc20120104="Actual I/O Mode" +ALIASES += ref_rfc20111119="New public functions to handle comparison" +ALIASES += ref_rfc20110825="Merging Named Datatypes in H5Ocopy()" +ALIASES += ref_rfc20110811="Expanding the HDF5 Hyperslab Selection Interface" +ALIASES += ref_rfc20110726="HDF5 File Space Allocation and Aggregation" +ALIASES += ref_rfc20110614=" Refactor h5dump to Improve Maintenance" +ALIASES += ref_rfc20110329="Support External Link Open File Cache in HDF5 Tools" +ALIASES += ref_rfc20110118="h5diff Attribute Comparisons" +ALIASES += ref_rfc20101122="SWMR Timeouts" +ALIASES += ref_rfc20101104="Caching Files Opened Through External Links" +ALIASES += ref_rfc20101018="HDF5 File and Object Comparison Specification" +ALIASES += ref_rfc20100902="h5edit – An HDF5 File Editing Tool" +ALIASES += ref_rfc20100727="Reserved Characters for HDF5 Applications" +ALIASES += ref_rfc20100726="High-Level HDF5 API routines for HPC Applications" +ALIASES += ref_rfc20100511="h5diff – Exclude Object(s) from Comparison" +ALIASES += ref_rfc20100422="Generating attributes into an object with a tool" +ALIASES += ref_rfc20100312="Supporting HDF5 1.8 in HDF5 Command Line Tools" +ALIASES += ref_rfc20091218="Supporting soft-link and external-link for h5diff" +ALIASES += ref_rfc20090907="HDF5 Tools Library Functions" +ALIASES += ref_rfc20090612="Default EPSILON values for comparing floating point data" +ALIASES += ref_rfc20081218="Reporting of Non-Comparable Datasets by h5diff" +ALIASES += ref_rfc20081205="External Link Traversal Callback" +ALIASES += ref_rfc20081030="Setting Raw Data Chunk Cache Parameters in HDF5" +ALIASES += ref_rfc20080915="Performance Report for Free-space Manager" +ALIASES += ref_rfc20080904="Setting File Access Property List for accessing External Link" +ALIASES += ref_rfc20080728="Native Time Types in HDF5" +ALIASES += ref_rfc20080723="Special Values in HDF5" +ALIASES += ref_rfc20080301="Dynamic Transformations to HDF5 Data" +ALIASES += ref_rfc20080209="Using SVN branching to improve software development process at THG" +ALIASES += ref_rfc20080206="Maintaining the HISTORY.txt and RELEASE.txt files in HDF5" +ALIASES += ref_rfc20071111="Addressing HDF5 file corruption issue" +ALIASES += ref_rfc20071018="NaN detection in HDF5" +ALIASES += ref_rfc20070801="Metadata Journaling to Improve Crash Survivability" +ALIASES += ref_rfc20070413="API Compatibility Strategies for HDF5" +ALIASES += ref_rfc20070115="A \"Private\" Heap for HDF5" +ALIASES += ref_rfc20060623="Performance Comparison of Collective I/O and Independent I/O with Derived Datatypes" +ALIASES += ref_rfc20060604="h5stat tool" +ALIASES += ref_rfc20060505="Simple Performance Test on Fletcher32 Filter" +ALIASES += ref_rfc20060410="Requirement Specifications of an HDF5 File Format Validation Tool" +ALIASES += ref_rfc20060317="Proposed changes to the sec2 driver " +ALIASES += ref_rfc20060124="Mapping FITS data to HDF5" +ALIASES += ref_rfc20040811="Conversion Between Text and Datatype" ################################################################################ # The Usual Suspects diff --git a/doxygen/dox/About.dox b/doxygen/dox/About.dox index e60e0fbdc7a..b077bfe804c 100644 --- a/doxygen/dox/About.dox +++ b/doxygen/dox/About.dox @@ -23,7 +23,7 @@ Use Doxygen's here. +here. \subsection new_rm_entry Creating a New Reference Manual Entry @@ -33,7 +33,7 @@ Please refer to the \ref RMT for guidance on how to create a new reference manua For each HDF5 module, such as \Code{H5F}, there is an examples source file called \Code{H5*_examples.c}. For example, the \Code{H5F} API examples are located in - + H5F_examples.c. Examples are code blocks marked as Doxygen snippets. For example, the source code for the H5Fcreate() API sample is located between @@ -44,7 +44,7 @@ the //! \endverbatim comments in - + H5F_examples.c. Add a new API example by adding a new code block enclosed between matching @@ -80,7 +80,7 @@ See Doxygen's Custom Comman as a general reference. All custom commands for this project are located in the -aliases +aliases file in the doxygen subdirectory of the main HDF5 repo. @@ -91,7 +91,7 @@ ask for help if unsure! For ease of reference, we define custom commands for each RFC in the RFCs section of the -aliases +aliases file. For example the custom command \Code{ref_rfc20141210} can be used to insert a reference to "RFC: Virtual Object Layer". In other words, the markup \verbatim @@ -102,16 +102,16 @@ yields a clickable link: \ref_rfc20141210 To add a new RFC, add a custom command for the RFC to the -aliases +aliases file. The naming convention for the custom command is \Code{ref_rfcYYYYMMDD}, where \Code{YYYYMMDD} is the ID of the RFC. The URL is composed of the prefix \verbatim -https://docs.hdfgroup.org/hdf5/rfc/ +https://\RFCURL/ \endverbatim and the name of your RFC file, typically, a PDF file, i.e., the full URL would be \verbatim -https://docs.hdfgroup.org/hdf5/rfc/my_great_rfc_name.pdf +https://\RFCURL/my_great_rfc_name.pdf \endverbatim \subsection hosting How Do Updates and Changes Get Published? diff --git a/doxygen/dox/ExamplesAPI.dox b/doxygen/dox/ExamplesAPI.dox index 12b585bdeb2..80a3dc1d66a 100644 --- a/doxygen/dox/ExamplesAPI.dox +++ b/doxygen/dox/ExamplesAPI.dox @@ -27,236 +27,236 @@ Languages are C, Fortran, Java (JHI5), Java Object Package, Python (High Level), Set Space Allocation Time for Dataset -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_alloc.h5 -h5ex_d_alloc.tst -h5ex_d_alloc.ddl +h5ex_d_alloc.tst +h5ex_d_alloc.ddl Read / Write Dataset using Fletcher32 Checksum Filter -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_checksum.h5 -h5ex_d_checksum.tst -h5ex_d_checksum.ddl +h5ex_d_checksum.tst +h5ex_d_checksum.ddl Read / Write Chunked Dataset -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_chunk.h5 -h5ex_d_chunk.tst -h5ex_d_chunk.ddl +h5ex_d_chunk.tst +h5ex_d_chunk.ddl Read / Write Compact Dataset -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_compact.h5 -h5ex_d_compact.tst -h5ex_d_compact.ddl +h5ex_d_compact.tst +h5ex_d_compact.ddl Read / Write to External Dataset -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_extern.h5 -h5ex_d_extern.tst -h5ex_d_extern.ddl +h5ex_d_extern.tst +h5ex_d_extern.ddl Read / Write Dataset w/ Fill Value -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_fillval.h5 -h5ex_d_fillval.tst -h5ex_d_fillval.ddl +h5ex_d_fillval.tst +h5ex_d_fillval.ddl Read / Write GZIP Compressed Dataset -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_gzip.h5 -h5ex_d_gzip.tst -h5ex_d_gzip.ddl +h5ex_d_gzip.tst +h5ex_d_gzip.ddl Read / Write Data by Hyperslabs -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_hyper.h5 -h5ex_d_hyper.tst -h5ex_d_hyper.ddl +h5ex_d_hyper.tst +h5ex_d_hyper.ddl Read / Write Dataset with n-bit Filter -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_nbit.h5 -h5ex_d_nbit.tst -h5ex_d_nbit.ddl +h5ex_d_nbit.tst +h5ex_d_nbit.ddl Read / Write Integer Dataset -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_rdwrc.h5 -h5ex_d_rdwrc.tst -h5ex_d_rdwr.ddl +h5ex_d_rdwrc.tst +h5ex_d_rdwr.ddl Read / Write Dataset w/ Shuffle Filter and GZIP Compression -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_d_shuffle.h5 -h5ex_d_shuffle.tst -h5ex_d_shuffle.ddl +h5ex_d_shuffle.tst +h5ex_d_shuffle.ddl Read / Write Dataset using Scale-Offset Filter (float) -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_d_sofloat.h5 -h5ex_d_sofloat.tst -h5ex_d_sofloat.ddl +h5ex_d_sofloat.tst +h5ex_d_sofloat.ddl Read / Write Dataset using Scale-Offset Filter (integer) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_soint.h5 -h5ex_d_soint.tst -h5ex_d_soint.ddl +h5ex_d_soint.tst +h5ex_d_soint.ddl Read / Write Dataset using SZIP Compression -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_szip.h5 -h5ex_d_szip.tst -h5ex_d_szip.ddl +h5ex_d_szip.tst +h5ex_d_szip.ddl Read / Write Dataset using Data Transform Expression -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_transform.h5 -h5ex_d_transform.tst -h5ex_d_transform.ddl +h5ex_d_transform.tst +h5ex_d_transform.ddl Read / Write Unlimited Dimension Dataset -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_d_unlimadd.h5 -h5ex_d_unlimadd.tst -h5ex_d_unlimadd.ddl +h5ex_d_unlimadd.tst +h5ex_d_unlimadd.ddl Read / Write GZIP Compressed Unlimited Dimension Dataset -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_d_unlimgzip.h5 -h5ex_d_unlimgzip.tst -h5ex_d_unlimgzip.ddl +h5ex_d_unlimgzip.tst +h5ex_d_unlimgzip.ddl Read / Write / Edit Unlimited Dimension Dataset -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_d_unlimmod.h5 -h5ex_d_unlimmod.tst -h5ex_d_unlimmod.ddl +h5ex_d_unlimmod.tst +h5ex_d_unlimmod.ddl @@ -272,105 +272,105 @@ FORTRAN Create "compact-or-indexed" Format Groups -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_g_compact.h5 -h5ex_g_.tst -h5ex_g_compact1.ddl -h5ex_g_compact2.ddl +h5ex_g_.tst +h5ex_g_compact1.ddl +h5ex_g_compact2.ddl Track links in a Group by Creation Order -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_g_corder.h5 -h5ex_g_corder.tst +h5ex_g_corder.tst h5ex_g_corder.ddl Create / Open / Close a Group -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_g_create.h5 h5ex_g_create.tst -h5ex_g_create.ddl +h5ex_g_create.ddl Create Intermediate Groups -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_g_intermediate.h5 -h5ex_g_intermediate.tst +h5ex_g_intermediate.tst h5ex_g_intermediate.ddl Iterate over Groups w/ H5Literate -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_g_iterate.h5 -h5ex_g_iterate.tst +h5ex_g_iterate.tst h5ex_g_iterate.ddl Set Conditions to Convert between Compact and Dense Groups -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_g_phase.h5 -h5ex_g_phase.tst +h5ex_g_phase.tst h5ex_g_phase.ddl Recursively Traverse a File with H5Literate -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_g_traverse.h5 -h5ex_g_traverse.tst +h5ex_g_traverse.tst h5ex_g_traverse.ddl Recursively Traverse a File with H5Ovisit / H5Lvisit -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_g_visit.h5 -h5ex_g_visit.tst +h5ex_g_visit.tst h5ex_g_visit.ddl @@ -387,347 +387,346 @@ FORTRAN Read / Write Array (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_arrayatt.h5 -h5ex_t_arrayatt.tst -h5ex_t_arrayatt.ddl +h5ex_t_arrayatt.tst +h5ex_t_arrayatt.ddl Read / Write Array (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_array.h5 -h5ex_t_array.tst -h5ex_t_array.ddl +h5ex_t_array.ddl Read / Write Bitfield (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_bitatt.h5 -h5ex_t_bitatt.tst -h5ex_t_bitatt.ddl +h5ex_t_bitatt.tst +h5ex_t_bitatt.ddl Read / Write Bitfield (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_bit.h5 -h5ex_t_bit.tst -h5ex_t_bit.ddl +h5ex_t_bit.tst +h5ex_t_bit.ddl Read / Write Compound (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_cmpdatt.h5 -h5ex_t_cmpdatt.tst -h5ex_t_cmpdatt.ddl +h5ex_t_cmpdatt.tst +h5ex_t_cmpdatt.ddl Read / Write Compound (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_cmpd.h5 -h5ex_t_cmpd.tst -h5ex_t_cmpd.ddl +h5ex_t_cmpd.tst +h5ex_t_cmpd.ddl Commit Named Datatype and Read Back -C +C FORTRAN -Java +Java JavaObj MATLAB PyHigh PyLow h5ex_t_commit.h5 -h5ex_t_commit.tst -h5ex_t_commit.ddl +h5ex_t_commit.tst +h5ex_t_commit.ddl Convert Between Datatypes in Memory -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_convert.h5 -h5ex_t_convert.tst -h5ex_t_convert.ddl +h5ex_t_convert.tst +h5ex_t_convert.ddl Read / Write Complex Compound (Attribute) -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_cpxcmpdatt.h5 -h5ex_t_cpxcmpdatt.tst -h5ex_t_cpxcmpdatt.ddl +h5ex_t_cpxcmpdatt.tst +h5ex_t_cpxcmpdatt.ddl Read / Write Complex Compound (Dataset) -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_cpxcmpd.h5 -h5ex_t_cpxcmpd.tst -h5ex_t_cpxcmpd.ddl +h5ex_t_cpxcmpd.tst +h5ex_t_cpxcmpd.ddl Read / Write Enumerated (Attribute) -C -FORTRAN +C +FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_enumatt.h5 -h5ex_t_enumatt.tst -h5ex_t_enumatt.ddl +h5ex_t_enumatt.tst +h5ex_t_enumatt.ddl Read / Write Enumerated (Dataset) -C -FORTRAN +C +FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_enum.h5 -h5ex_t_enum.tst -h5ex_t_enum.ddl +h5ex_t_enum.tst +h5ex_t_enum.ddl Read / Write Floating Point (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_floatatt.h5 -h5ex_t_floatatt.tst -h5ex_t_floatatt.ddl +h5ex_t_floatatt.tst +h5ex_t_floatatt.ddl Read / Write Floating Point (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_float.h5 -h5ex_t_float.tst -h5ex_t_float.ddl +h5ex_t_float.tst +h5ex_t_float.ddl Read / Write Integer Datatype (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_intatt.h5 -h5ex_t_intatt.tst -h5ex_t_intatt.ddl +h5ex_t_intatt.tst +h5ex_t_intatt.ddl Read / Write Integer Datatype (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_int.h5 -h5ex_t_int.tst -h5ex_t_int.ddl +h5ex_t_int.tst +h5ex_t_int.ddl Read / Write Object References (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_objrefatt.h5 -h5ex_t_objrefatt.tst -h5ex_t_objrefatt.ddl +h5ex_t_objrefatt.tst +h5ex_t_objrefatt.ddl Read / Write Object References (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_objref.h5 -h5ex_t_objref.tst -h5ex_t_objref.ddl +h5ex_t_objref.tst +h5ex_t_objref.ddl Read / Write Opaque (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_opaqueatt.h5 -h5ex_t_opaqueatt.tst -h5ex_t_opaqueatt.ddl +h5ex_t_opaqueatt.tst +h5ex_t_opaqueatt.ddl Read / Write Opaque (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_opaque.h5 -h5ex_t_opaque.tst -h5ex_t_opaque.ddl +h5ex_t_opaque.tst +h5ex_t_opaque.ddl Read / Write Region References (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_regrefatt.h5 -h5ex_t_regrefatt.tst -h5ex_t_regrefatt.ddl +h5ex_t_regrefatt.tst +h5ex_t_regrefatt.ddl Read / Write Region References (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_regref.h5 -h5ex_t_regref.tst -h5ex_t_regref.ddl +h5ex_t_regref.tst +h5ex_t_regref.ddl Read / Write String (Attribute) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_stringatt.h5 -h5ex_t_stringatt.tst -h5ex_t_stringatt.ddl +h5ex_t_stringatt.tst +h5ex_t_stringatt.ddl Read / Write String (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_string.h5 -h5ex_t_string.tst -h5ex_t_string.ddl +h5ex_t_string.tst +h5ex_t_string.ddl Read / Write Variable Length (Attribute) -C -FORTRAN +C +FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_vlenatt.h5 -h5ex_t_vlenatt.tst -h5ex_t_vlenatt.ddl +h5ex_t_vlenatt.tst +h5ex_t_vlenatt.ddl Read / Write Variable Length (Dataset) -C -FORTRAN +C +FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_vlen.h5 -h5ex_t_vlen.tst -h5ex_t_vlen.ddl +h5ex_t_vlen.tst +h5ex_t_vlen.ddl Read / Write Variable Length String (Attribute) -C -FORTRAN +C +FORTRAN Java JavaObj MATLAB PyHigh PyLow h5ex_t_vlstringatt.h5 -h5ex_t_vlstringatt.tst -h5ex_t_vlstringatt.ddl +h5ex_t_vlstringatt.tst +h5ex_t_vlstringatt.ddl Read / Write Variable Length String (Dataset) -C -FORTRAN -Java +C +FORTRAN +Java JavaObj MATLAB PyHigh PyLow h5ex_t_vlstring.h5 -h5ex_t_vlstring.tst -h5ex_t_vlstring.ddl +h5ex_t_vlstring.tst +h5ex_t_vlstring.ddl @@ -743,92 +742,92 @@ FORTRAN Read / Write Dataset using Blosc Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_blosc.h5 -h5ex_d_blosc.tst -h5ex_d_blosc.ddl +h5ex_d_blosc.h5 +h5ex_d_blosc.tst +h5ex_d_blosc.ddl Read / Write Dataset using Bit Shuffle Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_bshuf.h5 -h5ex_d_bshuf.tst -h5ex_d_bshuf.ddl +h5ex_d_bshuf.h5 +h5ex_d_bshuf.tst +h5ex_d_bshuf.ddl Read / Write Dataset using BZip2 Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_bzip2.h5 -h5ex_d_bzip2.tst -h5ex_d_bzip2.ddl +h5ex_d_bzip2.h5 +h5ex_d_bzip2.tst +h5ex_d_bzip2.ddl Read / Write Dataset using JPEG Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_jpeg.h5 -h5ex_d_jpeg.tst -h5ex_d_jpeg.ddl +h5ex_d_jpeg.h5 +h5ex_d_jpeg.tst +h5ex_d_jpeg.ddl Read / Write Dataset using LZ4 Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_lz4.h5 -h5ex_d_lz4.tst -h5ex_d_lz4.ddl +h5ex_d_lz4.h5 +h5ex_d_lz4.tst +h5ex_d_lz4.ddl Read / Write Dataset using LZF Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_lzf.h5 -h5ex_d_lzf.tst -h5ex_d_lzf.ddl +h5ex_d_lzf.h5 +h5ex_d_lzf.tst +h5ex_d_lzf.ddl Read / Write Dataset using MAFISC Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_mafisc.h5 -h5ex_d_mafisc.tst -h5ex_d_mafisc.ddl +h5ex_d_mafisc.h5 +h5ex_d_mafisc.tst +h5ex_d_mafisc.ddl Read / Write Dataset using ZFP Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_zfp.h5 -h5ex_d_zfp.tst -h5ex_d_zfp.ddl +h5ex_d_zfp.h5 +h5ex_d_zfp.tst +h5ex_d_zfp.ddl Read / Write Dataset using ZStd Compression -C +C FORTRAN Java JavaObj MATLAB PyHigh PyLow -h5ex_d_zstd.h5 -h5ex_d_zstd.tst -h5ex_d_zstd.ddl +h5ex_d_zstd.h5 +h5ex_d_zstd.tst +h5ex_d_zstd.ddl @@ -842,66 +841,66 @@ FORTRAN Create/Read/Write an Attribute -Java +Java JavaObj -HDF5AttributeCreate.txt +HDF5AttributeCreate.txt Create Datasets -Java +Java JavaObj -HDF5DatasetCreate.txt +HDF5DatasetCreate.txt Read/Write Datasets -Java +Java JavaObj -HDF5DatasetRead.txt +HDF5DatasetRead.txt Create an Empty File -Java +Java JavaObj -HDF5FileCreate.txt +HDF5FileCreate.txt Retrieve the File Structure -Java +Java JavaObj -HDF5FileStructure.txt +HDF5FileStructure.txt Create Groups -Java +Java JavaObj -HDF5GroupCreate.txt +HDF5GroupCreate.txt Select a Subset of a Dataset -Java +Java JavaObj -HDF5SubsetSelect.txt +HDF5SubsetSelect.txt Create Two Datasets Within Groups -Java +Java JavaObj -HDF5GroupDatasetCreate.txt +HDF5GroupDatasetCreate.txt @@ -917,8 +916,8 @@ FORTRAN Creating and Accessing a File -C -FORTRAN +C +FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -927,8 +926,8 @@ FORTRAN Creating and Accessing a Dataset -C -FORTRAN +C +FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -937,8 +936,8 @@ FORTRAN Writing and Reading Contiguous Hyperslabs -C -FORTRAN +C +FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -947,8 +946,8 @@ FORTRAN Writing and Reading Regularly Spaced Data Hyperslabs -C -FORTRAN +C +FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -957,8 +956,8 @@ FORTRAN Writing and Reading Pattern Hyperslabs -C -FORTRAN +C +FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -967,8 +966,8 @@ FORTRAN Writing and Reading Chunk Hyperslabs -C -FORTRAN +C +FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -977,7 +976,7 @@ FORTRAN Using the Subfiling VFD to Write a File Striped Across Multiple Subfiles -C +C FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -986,7 +985,7 @@ FORTRAN Write to Datasets with Filters Applied -C +C FORTRAN MATLAB PyHigh PyLow ph5_.h5 @@ -995,7 +994,7 @@ FORTRAN Collectively Write Datasets with Filters and Not All Ranks have Data -C +C FORTRAN MATLAB PyHigh PyLow ph5_.h5 diff --git a/doxygen/dox/GettingStarted.dox b/doxygen/dox/GettingStarted.dox index e48dbab543c..aa81ca28744 100644 --- a/doxygen/dox/GettingStarted.dox +++ b/doxygen/dox/GettingStarted.dox @@ -71,8 +71,8 @@ A brief introduction to Parallel HDF5. If you are new to HDF5 please see the @re HDF5-1.10 New Features -\li Introduction to the Virtual Dataset - VDS -\li Introduction to Single-Writer/Multiple-Reader (SWMR) +\li \ref VDS +\li \ref SWMR diff --git a/doxygen/dox/Glossary.dox b/doxygen/dox/Glossary.dox index 9ccd27d2166..6822cc9bbd7 100644 --- a/doxygen/dox/Glossary.dox +++ b/doxygen/dox/Glossary.dox @@ -71,7 +71,13 @@
Compact layout
-
+
+

A dataset storage layout where the dataset elements are stored in the + object header of the dataset. This layout is suitable for very small + datasets that can easily fit in the object header.

+

Compact layout can improve storage and access performance for files + that have many very small datasets.

+
@@ -174,17 +180,20 @@ and an HDF5 path name in that file.
File format
-
+
HDF5 file format refers to the structure and organization of the HDF5 data + being stored within the file.
Fill value
-
+
A default value assigned to data elements that have not yet been written.
Filter
-
+
+ Filters are optional stages that can be inserted in the data pipeline to implement + compression and error checking. User applications may also add custom filters.
\section GLS_G G diff --git a/doxygen/dox/IntroHDF5.dox b/doxygen/dox/IntroHDF5.dox index 4a00b80b2f8..50253ab4573 100644 --- a/doxygen/dox/IntroHDF5.dox +++ b/doxygen/dox/IntroHDF5.dox @@ -608,8 +608,7 @@ on the HDF-EOS Tools and Information Center pag \section secHDF5Examples Examples \li \ref LBExamples \li \ref ExAPI -\li Examples in the Source Code -\li Other Examples +\li Examples in the Source Code \section secHDF5ExamplesCompile How To Compile For information on compiling in C, C++ and Fortran, see: \ref LBCompiling @@ -618,10 +617,10 @@ For information on compiling in C, C++ and Fortran, see: \ref LBCompiling IDL, MATLAB, and NCL Examples for HDF-EOS Examples of how to access and visualize NASA HDF-EOS files using IDL, MATLAB, and NCL. -Miscellaneous Examples +Miscellaneous Examples These (very old) examples resulted from working with users, and are not fully tested. Most of them are in C, with a few in Fortran and Java. -Using Special Values +Using Special Values These examples show how to create special values in an HDF5 application. */ diff --git a/doxygen/dox/IntroParExamples.dox b/doxygen/dox/IntroParExamples.dox index 9d148f59d64..cdab44f35a5 100644 --- a/doxygen/dox/IntroParExamples.dox +++ b/doxygen/dox/IntroParExamples.dox @@ -95,7 +95,7 @@ Below is the example program:
-hyperslab_by_row.c +hyperslab_by_row.c
@@ -205,7 +205,7 @@ Below is the F90 example program which illustrates how to write contiguous hyper
-hyperslab_by_col.F90 +hyperslab_by_col.F90
@@ -275,7 +275,7 @@ Below is an example program for writing hyperslabs by column in Parallel HDF5:
-hyperslab_by_col.c +hyperslab_by_col.c
@@ -346,7 +346,7 @@ Below is the example program for writing hyperslabs by column in Parallel HDF5:
-hyperslab_by_row.F90 +hyperslab_by_row.F90
@@ -431,12 +431,12 @@ Below are example programs for writing hyperslabs by pattern in Parallel HDF5:
-hyperslab_by_pattern.c +hyperslab_by_pattern.c
-hyperslab_by_pattern.F90 +hyperslab_by_pattern.F90
@@ -530,12 +530,12 @@ Below are example programs for writing hyperslabs by pattern in Parallel HDF5:
-hyperslab_by_chunk.c +hyperslab_by_chunk.c
-hyperslab_by_chunk.F90 +hyperslab_by_chunk.F90
diff --git a/doxygen/dox/IntroParHDF5.dox b/doxygen/dox/IntroParHDF5.dox index 0e30249f50d..414a186af89 100644 --- a/doxygen/dox/IntroParHDF5.dox +++ b/doxygen/dox/IntroParHDF5.dox @@ -145,8 +145,8 @@ Following is example code for creating an access template in HDF5: \endcode The following example programs create an HDF5 file using Parallel HDF5: -C: file_create.c -F90: file_create.F90 +C: file_create.c +F90: file_create.F90 \subsection subsec_pintro_create_dset Creating and Accessing a Dataset with PHDF5 @@ -226,8 +226,8 @@ The following code demonstrates a collective write using Parallel HDF5: \endcode The following example programs create an HDF5 dataset using Parallel HDF5: -C: dataset.c -F90: dataset.F90 +C: dataset.c +F90: dataset.F90 \subsubsection subsec_pintro_hyperslabs Hyperslabs @@ -264,7 +264,6 @@ HDF5 by contiguous hyperslab, by regularly spaced data in a column/row, by patte -
Navigate back: \ref index "Main" / \ref GettingStarted diff --git a/doxygen/dox/LearnBasics.dox b/doxygen/dox/LearnBasics.dox index 2847be7f778..bbdf4224c09 100644 --- a/doxygen/dox/LearnBasics.dox +++ b/doxygen/dox/LearnBasics.dox @@ -59,7 +59,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create a file -C Fortran C++ Java Python +C Fortran C++ Java Python @@ -67,7 +67,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create a dataset -C Fortran C++ Java Python +C Fortran C++ Java Python @@ -75,7 +75,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Read and write to a dataset -C Fortran C++ Java Python +C Fortran C++ Java Python @@ -83,7 +83,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create an attribute -C Fortran C++ Java Python +C Fortran C++ Java Python @@ -91,7 +91,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create a group -C Fortran C++ Java Python +C Fortran C++ Java Python @@ -99,7 +99,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create groups in a file using absolute and relative paths -C Fortran C++ Java Python +C Fortran C++ Java Python @@ -107,7 +107,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create datasets in a group -C Fortran C++ Java Python +C Fortran C++ Java Python @@ -115,7 +115,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create a file and dataset and select/read a subset from the dataset -C Fortran C++ Java Python +C Fortran C++ Java Python Also see examples to Write by row (and column) below. @@ -123,7 +123,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create an extendible (unlimited dimension) dataset -C Fortran C++ Java Python +C Fortran C++ Java Python Also see examples to Extend by row (and column) below @@ -131,7 +131,7 @@ These examples (C, C++, Fortran) are provided in the HDF5 source code and (Unix) Create a chunked and compressed dataset -C Fortran C++ Java Python +C Fortran C++ Java Python diff --git a/doxygen/dox/LearnBasics2.dox b/doxygen/dox/LearnBasics2.dox index ed2810c59bf..8eda57bc0c2 100644 --- a/doxygen/dox/LearnBasics2.dox +++ b/doxygen/dox/LearnBasics2.dox @@ -468,7 +468,7 @@ If the offset were 1x1 (instead of 1x2), then the selection can be made: The selections above were tested with the -h5_subsetbk.c +h5_subsetbk.c example code. The memory dataspace was defined as one-dimensional. \subsection subsecLBDsetSubRWProgRem Remarks diff --git a/doxygen/dox/LearnBasics3.dox b/doxygen/dox/LearnBasics3.dox index 67a4b12e106..ce907d4f6a5 100644 --- a/doxygen/dox/LearnBasics3.dox +++ b/doxygen/dox/LearnBasics3.dox @@ -156,9 +156,9 @@ Specifically look at the \ref ExAPI. There are examples for different languages. The C example to create a chunked dataset is: -h5ex_d_chunk.c +h5ex_d_chunk.c The C example to create a compact dataset is: -h5ex_d_compact.c +h5ex_d_compact.c \section secLBDsetLayoutChange Changing the Layout after Dataset Creation The dataset layout is a Dataset Creation Property List. This means that once the dataset has been @@ -166,8 +166,8 @@ created the dataset layout cannot be changed. The h5repack utility can be used t to a new with a new layout. \section secLBDsetLayoutSource Sources of Information -Chunking in HDF5 -(See the documentation on Advanced Topics in HDF5) +Chunking in HDF5 +(See the documentation on Advanced Topics in HDF5) \see \ref sec_plist in the HDF5 \ref UG.
@@ -184,7 +184,7 @@ certain initial dimensions, then to later increase the size of any of the initia HDF5 requires you to use chunking to define extendible datasets. This makes it possible to extend datasets efficiently without having to excessively reorganize storage. (To use chunking efficiently, -be sure to see the advanced topic, Chunking in HDF5.) +be sure to see the advanced topic, Chunking in HDF5.) The following operations are required in order to extend a dataset: \li Declare the dataspace of the dataset to have unlimited dimensions for all dimensions that might eventually be extended. @@ -224,7 +224,7 @@ Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics \section secLBComDsetCreate Creating a Compressed Dataset HDF5 requires you to use chunking to create a compressed dataset. (To use chunking efficiently, -be sure to see the advanced topic, Chunking in HDF5.) +be sure to see the advanced topic, Chunking in HDF5.) The following operations are required in order to create a compressed dataset: \li Create a dataset creation property list. @@ -294,12 +294,12 @@ Specifically look at the \ref ExAPI. There are examples for different languages, where examples of using #H5Literate and #H5Ovisit/#H5Lvisit are included. The h5ex_g_traverse example traverses a file using H5Literate: -\li C: h5ex_g_traverse.c -\li F90: h5ex_g_traverse_F03.f90 +\li C: h5ex_g_traverse.c +\li F90: h5ex_g_traverse_F03.f90 The h5ex_g_visit example traverses a file using H5Ovisit and H5Lvisit: -\li C: h5ex_g_visit.c -\li F90: h5ex_g_visit_F03.f90 +\li C: h5ex_g_visit.c +\li F90: h5ex_g_visit_F03.f90
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics @@ -693,7 +693,7 @@ did = H5Dopen (file_id, "/foo/boo/moo"); /* absolute path */
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics -@page LBCompiling Compiling HDF5 Applications +/** @page LBCompiling Compiling HDF5 Applications Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics
@@ -1003,11 +1003,9 @@ There are log files for the configure, test, and build.
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics -@page LBTraining Training Videos -Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics -
+*/ -Training Videos +/ref LBTraining
Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics diff --git a/doxygen/dox/LearnHDFView.dox b/doxygen/dox/LearnHDFView.dox index 2916db841e6..2f0a0782e60 100644 --- a/doxygen/dox/LearnHDFView.dox +++ b/doxygen/dox/LearnHDFView.dox @@ -7,8 +7,8 @@ This tutorial enables you to get a feel for HDF5 by using the HDFView browser. I any programming experience. \section sec_learn_hv_install HDFView Installation -\li Download and install HDFView. It can be downloaded from the Download HDFView page. -\li Obtain the storm1.txt text file, used in the tutorial. +\li Download and install HDFView. It can be downloaded from the Download HDFView page. +\li Obtain the storm1.txt text file, used in the tutorial. \section sec_learn_hv_begin Begin Tutorial Once you have HDFView installed, bring it up and you are ready to begin the tutorial. @@ -113,11 +113,11 @@ Datatype information as is):
  • Double left click on the Storm dataset in the tree view. A window with an empty spreadsheet pops open.
    • -
  • Copy the data from the storm1.txt file into the dataset. +
  • Copy the data from the storm1.txt file into the dataset. -If you downloaded storm1.txt, +If you downloaded storm1.txt, then click on the Import/Export Data menu and select Import Data from -> Text File. -Specify a location, select storm1.txt +Specify a location, select storm1.txt and click on the Open button. Answer Yes in the dialog box that pops up (which asks if you wish to paste the selected data). @@ -246,7 +246,7 @@ in the file). Please note that the chunk sizes used in this topic are for demonstration purposes only. For information on chunking and specifying an appropriate chunk size, see the -Chunking in HDF5 documentation. +Chunking in HDF5 documentation. Also see the HDF5 Tutorial topic on \ref secLBComDsetCreate.
      @@ -317,8 +317,8 @@ You will see the Another Storm dataset in the Image group: -
    • Copy the data from the storm1.txt file into the dataset. (See the previous topic for copying -storm1.txt into a dataset.)
      • +
    • Copy the data from the storm1.txt file into the dataset. (See the previous topic for copying +storm1.txt into a dataset.)
    • Table -> Close, and save the data.
    • Right click on Another Storm, and select Open As.
    • Select the Image button in the Dataset Selection window that pops up. Click the Ok button at the @@ -375,7 +375,7 @@ Add data to the Storm Image dataset as was shown previously:
    • Right click on Storm Image, and select Open As to open the Dataset Selection window.
    • Click on the Spreadsheet button at the top left of the Dataset Selection window to view the image as a spreadsheet.
      • -
    • Copy the data from the storm1.txt file into the dataset.
      • +
    • Copy the data from the storm1.txt file into the dataset.
    • Close the dataset and save the data.
    • diff --git a/doxygen/dox/Overview.dox b/doxygen/dox/Overview.dox index 3bb4e17f15e..9609faaed66 100644 --- a/doxygen/dox/Overview.dox +++ b/doxygen/dox/Overview.dox @@ -23,11 +23,11 @@ documents cover a mix of tasks, concepts, and reference, to help a specific \par Versions Version-specific documentation (see the version in the title area) can be found here: - - HDF5 develop branch - - HDF5 1.14.x (this site) - - HDF5 1.12.x - - HDF5 1.10.x - - HDF5 1.8.x + - HDF5 develop branch + - HDF5 1.14.x (this site) + - HDF5 1.12.x + - HDF5 1.10.x + - HDF5 1.8.x \par Search If you are looking for a specific function, constant, type, etc., use the diff --git a/doxygen/dox/TechnicalNotes.dox b/doxygen/dox/TechnicalNotes.dox index 7edf0a0abf7..b1c809417f0 100644 --- a/doxygen/dox/TechnicalNotes.dox +++ b/doxygen/dox/TechnicalNotes.dox @@ -3,12 +3,14 @@ \li \ref api-compat-macros \li \ref APPDBG \li \ref FMTDISC +\li \ref FILEIMGOPS \li \ref FILTER \li \ref IOFLOW \li \ref TNMDC \li \ref MT \li \ref SWMR \li \ref VDS +\li \ref RELVERSION \li \ref VFL */ @@ -25,6 +27,12 @@ */ +/** \page RELVERSION HDF5 Library Release Version Numbers + +\htmlinclude LibraryReleaseVersionNumbers.html + +*/ + /** \page VFL HDF5 Virtual File Layer \htmlinclude VFL.html @@ -37,6 +45,12 @@ */ +/** \page FILEIMGOPS HDF5 File Image Operations + +\htmlinclude FileImageOps.html + +*/ + /** \page FILTER HDF5 Filters \htmlinclude Filters.html @@ -60,3 +74,4 @@ \htmlinclude intro_VDS.html */ + diff --git a/doxygen/dox/TrainingVideos.dox b/doxygen/dox/TrainingVideos.dox new file mode 100644 index 00000000000..be5f557b683 --- /dev/null +++ b/doxygen/dox/TrainingVideos.dox @@ -0,0 +1,48 @@ +/** @page LBTraining Training Videos +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics +
    + +
    + + + + +
    [**HDF5 Introduction**](https://youtu.be/S74Kc8QYDac)
    + +## Core Topics + + + + + + + + + + + + + + + + + +
    [**Data Model and Basic Usage, Core Topic #1**](https://youtu.be/vKuthH200eI)[**Datasets, Core Topic #2**](https://youtu.be/2oDXgecMMfo)[**Attributes, Core Topic #3**](https://youtu.be/d_P6LhmzmWQ)
    [**Groups and Links, Core Topic #4**](https://youtu.be/29J_FxPEDxo)[**Discovering File Structure, Core Topic #5**](https://youtu.be/rIl1Gz8zh-8)[**Partial IO, Core Topic #6**](https://youtu.be/RuzwPCFGBRE)
    [**Compound Datatype, Core Topic #7**](https://youtu.be/NXWwjrPdACY)
    + +## Advanced Topics: + + + + + + + +
    [**Dataset Storage Layouts, Advanced Topic #1**](https://youtu.be/mOTpu5KDpj8)[**Using Compression and Filters, Advanced Topic #2**](https://youtu.be/TvnDV-U9T4k)[**Using Command Line Tools, Advanced Topic #3**](https://youtu.be/CPIO1lmRkdM)
    + +
    +Navigate back: \ref index "Main" / \ref GettingStarted / \ref LearnBasics + +*/ + + + diff --git a/doxygen/dox/UsersGuide.dox b/doxygen/dox/UsersGuide.dox index 3dd26f1a40a..ca478e41ec2 100644 --- a/doxygen/dox/UsersGuide.dox +++ b/doxygen/dox/UsersGuide.dox @@ -342,7 +342,7 @@ These documents provide additional information for the use and tuning of specifi -

    Chunking in HDF5

    +

    Chunking in HDF5

    Structuring the use of chunking and tuning it for performance.

    @@ -350,7 +350,7 @@ These documents provide additional information for the use and tuning of specifi -

    Using the Direct Chunk Write Function

    +

    Using the Direct Chunk Write Function

    Describes another way that chunks can be written to datasets.

    @@ -358,7 +358,7 @@ These documents provide additional information for the use and tuning of specifi -

    Copying Committed Datatypes with H5Ocopy

    +

    Copying Committed Datatypes with H5Ocopy

    Describes how to copy to another file a dataset that uses a committed datatype or an object with an attribute that uses a committed datatype so that the committed datatype in the destination file can be used by multiple objects.

    @@ -366,7 +366,7 @@ These documents provide additional information for the use and tuning of specifi -

    Metadata Caching in HDF5

    +

    Metadata Caching in HDF5

    Managing the HDF5 metadata cache and tuning it for performance.

    @@ -374,7 +374,7 @@ These documents provide additional information for the use and tuning of specifi -

    HDF5 Dynamically Loaded Filters

    +

    HDF5 Dynamically Loaded Filters

    Describes how an HDF5 application can apply a filter that is not registered with the HDF5 Library.

    @@ -382,7 +382,7 @@ These documents provide additional information for the use and tuning of specifi -

    HDF5 File Image Operations

    +

    HDF5 File Image Operations

    Describes how to work with HDF5 files in memory. Disk I/O is not required when file images are opened, created, read from, or written to.

    @@ -390,7 +390,7 @@ These documents provide additional information for the use and tuning of specifi -

    Modified Region Writes

    +

    Modified Region Writes

    Describes how to set write operations for in-memory files so that only modified regions are written to storage. Available when the Core (Memory) VFD is used.

    @@ -398,7 +398,7 @@ These documents provide additional information for the use and tuning of specifi -

    Using Identifiers

    +

    Using Identifiers

    Describes how identifiers behave and how they should be treated.

    @@ -406,7 +406,7 @@ These documents provide additional information for the use and tuning of specifi -

    Using UTF-8 Encoding in HDF5 Applications

    +

    Using UTF-8 Encoding in HDF5 Applications

    Describes the use of UTF-8 Unicode character encodings in HDF5 applications.

    @@ -414,7 +414,7 @@ These documents provide additional information for the use and tuning of specifi -

    Freeing Memory Allocated by the HDF5 Library

    +

    Freeing Memory Allocated by the HDF5 Library

    Describes how inconsistent memory management can cause heap corruption or resource leaks and possible solutions.

    @@ -422,7 +422,7 @@ These documents provide additional information for the use and tuning of specifi -

    HDF5 Glossary

    +

    @ref GLS

    A glossary of terms.

    diff --git a/doxygen/dox/VOLConnGuide.dox b/doxygen/dox/VOLConnGuide.dox index 7a03ab1590d..e30a48428a8 100644 --- a/doxygen/dox/VOLConnGuide.dox +++ b/doxygen/dox/VOLConnGuide.dox @@ -92,7 +92,7 @@ Public header Files you will need to be familiar with include: Many VOL connectors are listed on The HDF Group's VOL plugin registration page, located at: -Registered VOL Connectors. +Registered VOL Connectors. Not all of these VOL connectors are supported by The HDF Group and the level of completeness varies, but the connectors found there can serve as examples of working implementations @@ -195,7 +195,7 @@ contact help@hdfgroup.org for help with this. We name you've chosen will appear on the registered VOL connectors page. As noted above, registered VOL connectors will be listed at: -Registered VOL Connectors +Registered VOL Connectors A new \b conn_version field has been added to the class struct for 1.13. This field is currently not used by the library so its use is determined by the connector author. Best practices for this field will be determined diff --git a/doxygen/dox/ViewTools.dox b/doxygen/dox/ViewTools.dox index cef6b6f0ef8..828a8e9cc1a 100644 --- a/doxygen/dox/ViewTools.dox +++ b/doxygen/dox/ViewTools.dox @@ -48,15 +48,15 @@ Navigate back: \ref index "Main" / \ref GettingStarted \section secViewToolsCommandObtain Obtain Tools and Files (Optional) Pre-built binaries for Linux and Windows are distributed within the respective HDF5 binary release -packages, which can be obtained from the Download HDF5 page. +packages, which can be obtained from the Download HDF5 page. HDF5 files can be obtained from various places such as \ref HDF5Examples and HDF-EOS and Tools and Information Center. Specifically, the following examples are used in this tutorial topic: \li HDF5 Files created from compiling the \ref LBExamples \li HDF5 Files on the \ref ExAPI page -\li NPP JPSS files, SVM01_npp.. (gzipped) -and SVM09_npp.. (gzipped) -\li HDF-EOS OMI-Aura file +\li NPP JPSS files, SVM01_npp.. (gzipped) +and SVM09_npp.. (gzipped) +\li HDF-EOS OMI-Aura file \section secViewToolsCommandTutor Tutorial Topics A variety of command-line tools are included in the HDF5 binary distribution. There are tools to view, @@ -196,44 +196,29 @@ The following h5dump options can be helpful in viewing the content and structure Comment --n, --contents - -Displays a list of the objects in a file - -See @ref subsubsecViewToolsViewContent_h5dumpEx1 - +-n, --contents +Displays a list of the objects in a file +See @ref subsubsecViewToolsViewContent_h5dumpEx1 --n 1, --contents=1 - -Displays a list of the objects and attributes in a file - -See @ref subsubsecViewToolsViewAttr_h5dumpEx6 - +-n 1, --contents=1 +Displays a list of the objects and attributes in a file +See @ref subsubsecViewToolsViewAttr_h5dumpEx6 --H, --header - -Displays header information only (no data) - -See @ref subsubsecViewToolsViewContent_h5dumpEx2 - +-H, --header +Displays header information only (no data) +See @ref subsubsecViewToolsViewContent_h5dumpEx2 --A 0, --onlyattr=0 - -Suppresses the display of attributes - -See @ref subsubsecViewToolsViewContent_h5dumpEx2 - +-A 0, --onlyattr=0 +Suppresses the display of attributes +See @ref subsubsecViewToolsViewContent_h5dumpEx2 --N P, --any_path=P - -Displays any object or attribute that matches path P - -See @ref subsubsecViewToolsViewAttr_h5dumpEx6 - +-N P, --any_path=P +Displays any object or attribute that matches path P +See @ref subsubsecViewToolsViewAttr_h5dumpEx6 @@ -997,7 +982,7 @@ In other words, it is an array of four elements, in which each element is a 3 by This dataset is much more complex. Also note that subsetting cannot be done on Array datatypes. -See this section for more information on the Array datatype. +See this section for more information on the Array datatype. \subsubsection subsubsecViewToolsViewDtypes_objref Object Reference An Object Reference is a reference to an entire object (dataset, group, or named datatype). diff --git a/doxygen/dox/ViewToolsJPSS.dox b/doxygen/dox/ViewToolsJPSS.dox index 9c153956797..18c8ccefa0f 100644 --- a/doxygen/dox/ViewToolsJPSS.dox +++ b/doxygen/dox/ViewToolsJPSS.dox @@ -11,8 +11,8 @@ Navigate back: \ref index "Main" / \ref GettingStarted / \ref ViewToolsCommand This tutorial illustrates how to use the HDF5 tools to examine NPP files from the JPSS project. The following files are discussed: \code -SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 (gzipped file) -SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 (gzipped file) +SVM09_npp_d20120229_t0849107_e0854511_b01759_c20120229145452682127_noaa_ops.h5 (gzipped file) +SVM01_npp_d20130524_t1255132_e1256374_b08146_c20130524192048864992_noaa_ops.h5 (gzipped file) \endcode \section secViewToolsJPSSDeter Determining File Contents diff --git a/doxygen/dox/rm-template.dox b/doxygen/dox/rm-template.dox index f2e48d18d1f..003d5c4b862 100644 --- a/doxygen/dox/rm-template.dox +++ b/doxygen/dox/rm-template.dox @@ -2,9 +2,9 @@ We treat documentation like code and use Doxygen to -markup +markup comments in the code or create -stand-alone pages. +stand-alone pages. Every RM entry consists of a subset of the elements listed below. Not every RM entry warrants the full set. More is better, and we can, perhaps, distinguish diff --git a/doxygen/examples/FileFormat.html b/doxygen/examples/FileFormat.html index 30428e3cad2..eff1aead0a3 100644 --- a/doxygen/examples/FileFormat.html +++ b/doxygen/examples/FileFormat.html @@ -36,7 +36,7 @@ TABLE CAPTION STRONG { font-size: larger }

  • Background Reading:

    -
    HDF5 File Format Specification +
    HDF5 File Format Specification
    This describes the current HDF5 file format.
    diff --git a/doxygen/examples/FileImageOps.html b/doxygen/examples/FileImageOps.html new file mode 100644 index 00000000000..2409979ff7b --- /dev/null +++ b/doxygen/examples/FileImageOps.html @@ -0,0 +1,1610 @@ + + + + HDF5 File Image Operations + + + + + + + + +

    1. Introduction to HDF5 File Image Operations

    + +

    File image operations allow users to work with HDF5 files in memory in the same ways that users currently work with HDF5 files on disk. Disk I/O is not required when file images are opened, created, read from, or written to.

    +

    An HDF5 file image is an HDF5 file that is held in a buffer in main memory. Setting up a file image in memory involves using either a buffer in the file access property list or a buffer in the Core (aka Memory) file driver.

    +

    The advantage of working with a file in memory is faster access to the data.

    +

    The challenge of working with files in memory buffers is maximizing performance and minimizing memory footprint while working within the constraints of the property list mechanism. This should be a non-issue for small file images, but may be a major issue for large images.

    +

    If invoked with the appropriate flags, the H5LTopen_file_image() high level library call should deal with these challenges in most cases. However, some applications may require the programmer to address these issues directly.

    + +

    1.1. File Image Operations Function Summary

    +

    Functions used in file image operations are listed below.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    C FunctionPurposeSection
    H5Pset_file_imageSpecifies an initial file image2.1.1
    H5Pget_file_imageRetrieves a copy of the file image designated for a VFD to use as the initial contents of a file2.1.2
    H5Pset_file_image_callbacksManages file image buffer allocation, copying, reallocation, and release2.1.3
    H5Pget_file_image_callbacksObtains the current file image callbacks from a file access property list2.1.4
    H5Fget_file_imageProvides a simple way to retrieve a copy of the image of an existing, open file2.1.5
    H5LTopen_file_imageProvides a convenient way to open an initial file image with the Core VFD2.1.6
    + +

    1.2. Abbreviations

    +

    The following abbreviations are used in this document:

    + + + + + + + + + + + + + + + + + +
    AbbreviationExplanation
    FAPL or faplFile Access Property List. In code samples, fapl is used.
    VFDVirtual File Driver
    VFLVirtual File Layer
    + +

    1.3. Developer Prerequisites

    +

    Developers who use the file image operations described in this document should be proficient and experienced users of the HDF5 C Library APIs. More specifically, developers should have a working knowledge of property lists, callbacks, and virtual file drivers.

    + +

    1.4. Resources

    +

    See the following for more information.

    +

    The “RFC: File Image Operations” is the primary source for the information in this document.

    +

    The “Alternate File Storage Layouts and Low-level File Drivers” section is in “The HDF5 File” chapter of the HDF5 User’s Guide .

    +

    The H5P_SET_FAPL_CORE function call can be used to modify the file access property list so that the Memory virtual file driver, H5FD_CORE, is used. The Memory file driver is also known as the Core file driver.

    +

    Refer to the Virtual File Layer for more detail. A list of VFL Functions is provided below.

    +

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    C FunctionPurpose
    H5Pset_driverSets a file driver
    H5Pget_driver_infoReturns a pointer to file driver information
    H5FDregisterRegisters a new file driver as a member of the virtual file driver class
    H5FDunregisterRemoves a driver ID from the library
    H5FDopenOpens a file
    H5FDcloseCloses the file using the driver 'close' callback
    H5FDcmpCompares the keys of two files using the file driver callback if the files belong to the same driver, otherwise sort the files by driver class pointer value
    H5FDqueryQueries a VFL driver for its feature flags
    H5FDallocAllocates memory from the file
    H5FDfreeFrees format addresses in the file
    H5FDset_eoaSet the end-of-address marker for the file
    H5FDget_eoaReturns the address of the first byte after the last allocated memory in the file
    H5FDget_eofReturns the end-of-file address, which is the greater of the end-of-format address and the actual EOF marker
    H5FDreadReads bytes from the file beginning at the specified address according to the provided data transfer property list
    H5FDwriteWrites bytes to the file beginning at the specified address according to the provided data transfer property list
    H5FDflushNotifies driver to flush all cached data
    + +

    2. C API Call Syntax

    + +

    The C API function calls described in this chapter fall into two categories: low-level routines that are part of the main HDF5 C Library and one high-level routine that is part of the “lite” API in the high-level wrapper library. The high-level routine uses the low-level routines and presents frequently requested functionality conveniently packaged for application developers’ use.

    + +

    2.1. Low-level C API Routines

    + +

    The purpose of this section is to describe the low-level C API routines that support file image operations. These routines allow an in-memory image of an HDF5 file to be opened without requiring file system I/O.

    +

    The basic approach to opening an in-memory image of an HDF5 file is to pass the image to the Core file driver, and then tell the Core file driver to open the file. We do this by using the H5Pget/set_file_image calls. These calls allow the user to specify an initial file image.

    +

    A potential problem with the H5Pget/set_file_image calls is the overhead of allocating and copying of large file image buffers. The callback routines enable application programs to avoid this problem. However, the use of these callbacks is complex and potentially hazardous: the particulars are discussed in the semantics and examples chapters below (see section 3.1 and section 4.1 respectively). Fortunately, use of the file image callbacks should seldom be necessary: the H5LTopen_file_image call should address most use cases.

    +

    The property list facility in HDF5 is employed in file image operations. This facility was designed for passing data, not consumable resources, into API calls. The peculiar ways in which the file image allocation callbacks may be used allows us to avoid extending the property list structure to handle consumable resources cleanly and to avoid constructing a new facility for the purpose.

    +

    The sub-sections below describe the low-level C APIs that are used with file image operations.

    + +

    2.1.1. H5Pset_file_image

    + +

    The H5Pset_file_image routine allows an application to provide an image for a file driver to use as the initial contents of the file. This call was designed initially for use with the Core VFD, but it can be used with any VFD that supports using an initial file image when opening a file. See the “Virtual File Driver Feature Flags” section for more information. Calling this routine makes a copy of the provided file image buffer. See the “H5Pset_file_image_callbacks” section for more information.

    +

    The signature of H5Pset_file_image is defined as follows:

    +

    herr_t H5Pset_file_image(hid_t fapl_id, void *buf_ptr, size_t buf_len) + +

    The parameters of H5Pset_file_image are defined as follows:

    +

    fapl_id contains the ID of the target file access property list. +buf_ptr supplies a pointer to the initial file image, or NULL if no initial file image is desired. +buf_len contains the size of the supplied buffer, or 0 if no initial image is desired. +If either the buf_len parameter is zero, or the buf_ptr parameter is NULL, no file image will be set in the FAPL, and any existing file image buffer in the FAPL will be released. If a buffer is released, the FAPL’s file image buf_len will be set to 0 and buf_ptr will be set to NULL.

    +

    Given the tight interaction between the file image callbacks and the file image, the file image callbacks in a property list cannot be changed while a file image is defined.

    +

    With properly constructed file image callbacks, it is possible to avoid actually copying the file image. The particulars of this are discussed in greater detail in the “C API Call Semantics” chapter and in the “Examples” chapter.

    + +

    2.1.2. H5Pget_file_image

    + +

    The H5Pget_file_image routine allows an application to retrieve a copy of the file image designated for a VFD to use as the initial contents of a file. This routine uses the file image callbacks (if defined) when allocating and loading the buffer to return to the application, or it uses malloc and memcpy if the callbacks are undefined. When malloc and memcpy are used, it will be the caller’s responsibility to discard the returned buffer via a call to free.

    +

    The signature of H5Pget_file_image is defined as follows:

    +

    herr_t H5Pget_file_image(hid_t fapl_id, void **buf_ptr_ptr, size_t *buf_len_ptr) + +

    The parameters of H5Pget_file_image are defined as follows:

    +

    fapl_id contains the ID of the target file access property list. +

    buf_ptr_ptr contains a NULL or a pointer to a void*. If buf_ptr_ptr is not NULL, on successful return, *buf_ptr_ptr will contain a pointer to a copy of the initial image provided in the last call to H5Pset_file_image for the supplied fapl_id. If no initial image has been set, *buf_ptr_ptr will be NULL.

    +

    buf_len_ptr contains a NULL or a pointer to size_t. If buf_len_ptr is not NULL, on successful return, *buf_len_ptr will contain the value of the buf_len parameter for the initial image in the supplied fapl_id. If no initial image is set, the value of *buf_len_ptr will be 0.

    + +

    As with H5Pset_file_image, appropriately defined file image callbacks can allow this function to avoid buffer allocation and memory copy operations.

    + +

    2.1.3. H5Pset_file_image_callbacks

    + +

    The H5Pset_file_image_callbacks API call exists to allow an application to control the management of file image buffers through user defined callbacks. These callbacks will be used in the management of file image buffers in property lists and in select file drivers. These routines are invoked when a new file image buffer is allocated, when an existing file image buffer is copied or resized, or when a file image buffer is released from use. From the perspective of the HDF5 Library, the operations of the image_malloc, image_memcpy, image_realloc, and image_free callbacks must be identical to those of the corresponding C standard library calls (malloc, memcpy, realloc, and free). While the operations must be identical, the file image callbacks have more parameters. The callbacks and their parameters are described below. The return values of image_malloc and image_realloc are identical to the return values of malloc and realloc. However, the return values of image_memcpy and image_free are different than the return values of memcpy and free: the return values of image_memcpy and image_free can also indicate failure. See the “File Image Callback Semantics” section for more information.

    +

    The signature of H5Pset_file_image_callbacks is defined as follows:

    + +

    typedef enum
+{
+  H5_FILE_IMAGE_OP_PROPERTY_LIST_SET,
+  H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY,
+  H5_FILE_IMAGE_OP_PROPERTY_LIST_GET,
+  H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE,
+  H5_FILE_IMAGE_OP_FILE_OPEN,
+  H5_FILE_IMAGE_OP_FILE_RESIZE,
+  H5_FILE_IMAGE_OP_FILE_CLOSE
+} H5_file_image_op_t;

    + +

    typedef struct
+{
+  void *(*image_malloc)(size_t size, H5_file_image_op_t file_image_op,
+        void *udata);
+  void *(*image_memcpy)(void *dest, const void *src, size_t size,
+        H5_file_image_op_t file_image_op, void *udata);
+  void *(*image_realloc)(void *ptr, size_t size,
+        H5_file_image_op_t file_image_op, void *udata);
+  herr_t (*image_free)(void *ptr, H5_file_image_op_t file_image_op,
+        void *udata);
+  void *(*udata_copy)(void *udata);
+  herr_t (*udata_free)(void *udata);
+  void *udata;
+} H5_file_image_callbacks_t;

    + +

    herr_t H5Pset_file_image_callbacks(hid_t fapl_id, + H5_file_image_callbacks_t *callbacks_ptr)

    + +The parameters of H5Pset_file_image_callbacks are defined as follows:

    + +
      +
    • fapl_id contains the ID of the target file access property list.
      • +
    • callbacks_ptr contains a pointer to an instance of the H5_file_image_callbacks_t structure.
      • +
    + +

    The fields of the H5_file_image_callbacks_t structure are defined as follows: + +

      +
    • image_malloc contains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library malloc() call. The parameters of the image_malloc callback are defined as follows:
      • +
        +
      • size contains the size in bytes of the image buffer to allocate.
        • +
      • file_image_op contains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2.
        • +
      • udata holds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
        • +
      + + Setting image_malloc to NULL indicates that the HDF5 Library should invoke the standard C library malloc() routine when allocating file image buffers.

      + +

    • image_memcpy contains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library memcpy() call except that it returns NULL on failure. Recall that the memcpy C Library routine is defined to return the dest parameter in all cases. The parameters of the image_memcpy callback are defined as follows:
      • +
        +
      • dest contains the address of the destination buffer.
        • +
      • src contains the address of the source buffer.
        • +
      • size contains the number of bytes to copy.
        • +
      • file_image_op contains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2.
        • +
      • udata holds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
        • +
      + + Setting image_memcpy to NULL indicates that the HDF5 Library should invoke the standard C library memcpy() routine when copying buffers.

      + +

    • image_realloc contains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library realloc() call. The parameters of the image_realloc callback are defined as follows:

      • +
        +
      • ptr contains the pointer to the buffer being reallocated.
        • +
      • size contains the desired size in bytes of the buffer after realloc.
        • +
      • file_image_op contains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2.
        • +
      • udata holds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
        • +
      + +Setting image_realloc to NULL indicates that the HDF5 Library should invoke the standard C library realloc() routine when resizing file image buffers.

      + +

    • image_free contains a pointer to a function with (from the perspective of HDF5) functionality identical to the standard C library free() call except that it will return 0 (SUCCEED) on success and -1 (FAIL) on failure. The parameters of the image_free callback are defined as follows:
      • +
        +
      • ptr contains the pointer to the buffer being released.
        • +
      • file_image_op contains one of the values of H5_file_image_op_t. These values indicate the operation being performed on the file image when this callback is invoked. Possible values for file_image_op are discussed in Table 2 .
        • +
      • udata holds the value passed in for the udata parameter to H5Pset_file_image_callbacks.
        • +
      + + Setting image_free to NULL indicates that the HDF5 Library should invoke the standard C library free() routine when releasing file image buffers.

      + +

    • udata_copy contains a pointer to a function that (from the perspective of HDF5) allocates a buffer of suitable size, copies the contents of the supplied udata into the new buffer, and returns the address of the new buffer. The function returns NULL on failure. This function is necessary if a non-NULL udata parameter is supplied, so that property lists containing the image callbacks can be copied. If the udata parameter (below) is NULL, then this parameter should be NULL as well. The parameter of the udata_copy callback is defined as follows:
      • +
        +
      • udata contains the pointer to the user data block being copied.
        • +

      + +

    • udata_free contains a pointer to a function that (from the perspective of HDF5) frees a user data block. This function is necessary if a non-NULL udata parameter is supplied so that property lists containing image callbacks can be discarded without a memory leak. If the udata parameter (below) is NULL, this parameter should be NULL as well. The parameter of the udata_free callback is defined as follows:
      • +
        +
      • udata contains the pointer to the user data block to be freed.
        • +
      + + udata_free returns 0 (SUCCEED) on success and -1 (FAIL) on failure.

      + +
    • udata contains a pointer value, potentially to user-defined data, that will be passed to the image_malloc, image_memcpy, image_realloc, and image_free callbacks.
      • +
    + +The semantics of the values that can be set for the file_image_op parameter to the above callbacks are described in the table below:

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Table 2. Values for the file_image_op parameter
    ValueComments
    H5_FILE_IMAGE_OP_PROPERTY_LIST_SETThis value is passed to the image_malloc and image_memcpy callbacks when an image buffer is being copied while being set in a FAPL
    H5_FILE_IMAGE_OP_PROPERTY_LIST_COPYThis value is passed to the image_malloc and image_memcpy callbacks when an image buffer is being copied when a FAPL is copied
    H5_FILE_IMAGE_OP_PROPERTY_LIST_GETThis value is passed to the image_malloc and image_memcpy callbacks when an image buffer is being copied while being retrieved from a FAPL
    H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSEThis value is passed to the image_free callback when an image buffer is being released during a FAPL close operation.
    H5_FILE_IMAGE_OP_FILE_OPENThis value is passed to the image_malloc and image_memcpy callbacks when an image buffer is copied during a file open operation. While the image being opened will typically be copied from a FAPL, this need not always be the case. An example of an exception is when the Core file driver takes its initial image from a file.
    H5_FILE_IMAGE_OP_FILE_RESIZEThis value is passed to the image_realloc callback when a file driver needs to resize an image buffer.
    H5_FILE_IMAGE_OP_FILE_CLOSEThis value is passed to the image_free callback when an image buffer is being released during a file close operation.
    + +

    In closing our discussion of H5Pset_file_image_callbacks(), we note the interaction between this call and the H5Pget/set_file_image() calls above: since the malloc, memcpy, and free callbacks defined in the instance of H5_file_image_callbacks_t are used by H5Pget/set_file_image(), H5Pset_file_image_callbacks() will fail if a file image is already set in the target property list.

    +

    For more information on writing the file image to disk, set the backing_store parameter. See the H5Pset_fapl_core entry in the HDF5 Reference Manual.

    + +

    2.1.4. H5Pget_file_image_callbacks

    +

    The H5Pget_file_image_callbacks routine is designed to obtain the current file image callbacks from a file access property list.

    +

    The signature of H5Pget_file_image_callbacks() is defined as follows:

    + +

    herr_t H5Pget_file_image_callbacks(hid_t fapl_id, + H5_file_image_callbacks_t *callbacks_ptr)

    + +The parameters of H5Pget_file_image_callbacks are defined as follows:

    + +
      +
    • fapl_id contains the ID of the target file access property list.
      • +
    • callbacks_ptr contains a pointer to an instance of the H5_file_image_callbacks_t structure. All fields should be initialized to NULL. See the “H5Pset_file_image_callbacks” section for more information on the H5_file_image_callbacks_t structure.
      • +
    + +Upon successful return, the fields of *callbacks_ptr shall contain values as defined below:

    +
      +
    • Upon successful return, callbacks_ptr->image_malloc will contain the pointer passed as the image_malloc field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call.
      • +
    • Upon successful return, callbacks_ptr->image_memcpy will contain the pointer passed as the image_memcpy field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call.
      • +
    • Upon successful return, callbacks_ptr->image_realloc will contain the pointer passed as the image_realloc field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call.
      • +
    • Upon successful return, callbacks_ptr->image_free_ptr will contain the pointer passed as the image_free field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call.
      • +
    • Upon successful return, callbacks_ptr->udata_copy will contain the pointer passed as the udata_copy field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call.
      • +
    • Upon successful return, callbacks_ptr-> udata_free will contain the pointer passed as the udata_free field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call.
      • +
    • Upon successful return, callbacks_ptr->udata will contain the pointer passed as the udata field of the instance of H5_file_image_callbacks_t pointed to by the callbacks_ptr parameter of the last call to H5Pset_file_image_callbacks() for the specified FAPL, or NULL if there has been no such call.
      • +
    + +

    2.1.5. Virtual File Driver Feature Flags

    + +Implementation of the H5Pget/set_file_image_callbacks() and H5Pget/set_file_image() function calls requires a pair of virtual file driver feature flags. The flags are H5FD_FEAT_ALLOW_FILE_IMAGE and H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS. Both of these are defined in H5FDpublic.h.

    + +

    The first flag, H5FD_FEAT_ALLOW_FILE_IMAGE, allows a file driver to indicate whether or not it supports file images. A VFD that sets this flag when its ‘query’ callback is invoked indicates that the file image set in the FAPL will be used as the initial contents of a file. Support for setting an initial file image is designed primarily for use with the Core VFD. However, any VFD can indicate support for this feature by setting the flag and copying the image in an appropriate way for the VFD (possibly by writing the image to a file and then opening the file). However, such a VFD need not employ the file image after file open time. In such cases, the VFD will not make an in-memory copy of the file image and will not employ the file image callbacks.

    + +

    File drivers that maintain a copy of the file in memory (only the Core file driver at present) can be constructed to use the initial image callbacks (if defined). Those that do must set the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag, the second flag, when their ‘query’ callbacks are invoked.

    + +

    Thus file drivers that set the H5FD_FEAT_ALLOW_FILE_IMAGE flag but not the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag may read the supplied image from the property list (if present) and use it to initialize the contents of the file. However, they will not discard the image when done, nor will they make any use of any file image callbacks (if defined).

    + +

    If an initial file image appears in a file allocation property list that is used in an H5Fopen() call, and if the underlying file driver does not set the H5FD_FEAT_ALLOW_FILE_IMAGE flag, then the open will fail.

    + +

    If a driver sets both the H5FD_FEAT_ALLOW_FILE_IMAGE flag and the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag, then that driver will allocate a buffer of the required size, copy the contents of the initial image buffer from the file access property list, and then open the copy as if it had just loaded it from file. If the file image allocation callbacks are defined, the driver shall use them for all memory management tasks. Otherwise it will use the standard malloc, memcpy, realloc, and free C library calls for this purpose.

    + +

    If the VFD sets the H5FD_FEAT_ALLOW_FILE_IMAGE flag, and an initial file image is defined by an application, the VFD should ensure that file creation operations (as opposed to file open operations) bypass use of the file image, and create a new, empty file.

    + +

    Finally, it is logically possible that a file driver would set the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flag, but not the H5FD_FEAT_ALLOW_FILE_IMAGE flag. While it is hard to think of a situation in which this would be desirable, setting the flags this way will not cause any problems: the two capabilities are logically distinct.

    + +

    2.1.6. H5Fget_file_image

    + +

    The purpose of the H5Fget_file_image routine is to provide a simple way to retrieve a copy of the image of an existing, open file. This routine can be used with files opened using the SEC2 (aka POSIX), STDIO, and Core (aka Memory) VFDs.

    + +

    The signature of H5Fget_file_image is defined as follows:

    +

    ssize_t H5Fget_file_image(hid_t file_id, void *buf_ptr, size_t buf_len)

    + +The parameters of H5Fget_file_image are defined as follows: + +
      +
    • file_id contains the ID of the target file.
      • +
    • buf_ptr contains a pointer to the buffer into which the image of the HDF5 file is to be copied. If buf_ptr is NULL, no data will be copied, but the return value will still indicate the buffer size required (or a negative value on error).
      • +
    • buf_len contains the size of the supplied buffer.
      • +
    + +

    If the return value of H5Fget_file_image is a positive value, then the value will be the length of buffer required to store the file image (in other words, the length of the file). A negative value might be returned if the file is too large to store in the supplied buffer or on failure.

    + +

    The current file size can be obtained via a call to H5Fget_filesize(). Note that this function returns the value of the end of file (EOF) and not the end of address space (EOA). While these values are frequently the same, it is possible for the EOF to be larger than the EOA. Since H5Fget_file_image() will only obtain a copy of the file from the beginning of the superblock to the EOA, it will be best to use H5Fget_file_image() to determine the size of the buffer required to contain the image.

    + +Other Design Considerations + +

    Here are some other notes regarding the design and implementation of H5Fget_file_image.

    +

    The H5Fget_file_image call should be part of the high-level library. However, a file driver agnostic implementation of the routine requires access to data structures that are hidden within the HDF5 Library. We chose to implement the call in the library proper rather than expose those data structures.

    +

    There is no reason why the H5Fget_file_image() API call could not work on files opened with any file driver. However, the Family, Multi, and Split file drivers have issues that make the call problematic. At present, files opened with the Family file driver are marked as being created with that file driver in the superblock, and the HDF5 Library refuses to open files so marked with any other file driver. This negates the purpose of the H5Fget_file_image() call. While this mark can be removed from the image, the necessary code is not trivial.

    +

    Thus we will not support the Family file driver in H5Fget_file_image() unless there is demand for it. Files created with the Multi and Split file drivers are also marked in the superblock. In addition, they typically use a very sparse address space. A sparse address space would require the use of an impractically large buffer for an image, and most of the buffer would be empty. So, we see no point in supporting the Multi and Split file drivers in H5Fget_file_image() under any foreseeable circumstances.

    + +

    2.2. High-level C API Routine

    + +

    The H5LTopen_file_image high-level routine encapsulates the capabilities of routines in the main HDF5 Library with conveniently accessible abstractions.

    + +

    2.2.1. H5LTopen_file_image

    + +

    The H5LTopen_file_image routine is designed to provide an easier way to open an initial file image with the Core VFD. Flags to H5LTopen_file_image allow for various file image buffer ownership policies to be requested. See the HDF5 Reference Manual for more information on high-level APIs.

    +

    The signature of H5LTopen_file_image is defined as follows:

    + +

    hid_t H5LTopen_file_image(void *buf_ptr, size_t buf_len, unsigned flags) + +

    The parameters of H5LTopen_file_image are defined as follows:

    +
      +
    • buf_ptr contains a pointer to the supplied initial image. A NULL value is invalid and will cause H5LTopen_file_image to fail.
      • +
    • buf_len contains the size of the supplied buffer. A value of 0 is invalid and will cause H5LTopen_file_image to fail.
      • +
    • flags contains a set of flags indicating whether the image is to be opened read/write, whether HDF5 is to take control of the buffer, and how long the application promises to maintain the buffer. Possible flags are described in the table below:
      • +
    + + + + + + + + + + + + + + + + + + + +
    Table 3. Flags for H5LTopen_file_image
    ValueComments
    H5LT_FILE_IMAGE_OPEN_RWIndicates that the HDF5 Library should open the image read/write instead of the default read-only.
    H5LT_FILE_IMAGE_DONT_COPY +
      +
    • Indicates that the HDF5 Library should not copy the file image buffer provided, but should use it directly. The HDF5 Library will release the file image when finished. The supplied buffer must have been allocated via a call to the standard C library malloc() or calloc() routines. The HDF5 Library will call free() to release the buffer. In the absence of this flag, the HDF5 Library will copy the buffer provided. The H5LT_FILE_IMAGE_DONT_COPY flag provides an application with the ability to “give ownership” of a file image buffer to the HDF5 Library.
      • +
    • The HDF5 Library will modify the buffer on write if the image is opened read/write and the H5LT_FILE_IMAGE_DONT_COPY flag is set.
      • +
    • The H5LT_FILE_IMAGE_DONT_RELEASE flag, see below, is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag is set
      • +
    +
    H5LT_FILE_IMAGE_DONT_RELEASE +
      +
    • Indicates that the HDF5 Library should not attempt to release the buffer when the file is closed. This implies that the application will tend to this detail and that the application will not discard the buffer until after the file image is closed.
      • +
    • Since there is no way to return a changed buffer base address to the application, and since realloc can change this value, calls to realloc() must be barred when this flag is set. As a result, any write that requires an increased buffer size will fail.
      • +
    • This flag is invalid unless the H5LT_FILE_IMAGE_DONT_COPY flag, see above, is set.
      • +
    • If the H5LT_FILE_IMAGE_DONT_COPY flag is set and this flag is not set, the HDF5 Library will release the file image buffer after the file is closed using the standard C library free() routine.
      • +
    • Using this flag and the H5LT_FILE_IMAGE_DONT_COPY flag provides a way for the application to specify a buffer that the HDF5 Library can use for opening and accessing as a file image while letting the application retain ownership of the buffer.
      • +
    +
    + +

    The following table is intended to summarize the semantics of the H5LT_FILE_IMAGE_DONT_COPY and H5LT_FILE_IMAGE_DONT_RELEASE flags (shown as “Don’t Copy Flag” and “Don’t Release Flag” respectively in the table):

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Table 4. Summary of Don’t Copy and Don’t Release Flag Actions
    Don’t Copy FlagDon’t Release FlagMake Copy of User Supplied BufferPass User Supplied Buffer to File DriverRelease User Supplied Buffer When DonePermit realloc of Buffer Used by File Driver
    FalseDon’t careTrueFalseFalseTrue
    TrueFalseFalseTrueTrueTrue
    TrueTrueFalseTrueFalseFalse
    + +

    The return value of H5LTopen_file_image will be a file ID on success or a negative value on failure. The file ID returned should be closed with H5Fclose.

    +

    Note that there is no way currently to specify a “backing store” file name in this definition of H5LTopen_image.

    + +

    3. C API Call Semantics

    +

    The purpose of this chapter is to describe some issues that developers should consider when using file image buffers, property lists, and callback APIs.

    + +

    3.1. File Image Callback Semantics

    +

    The H5Fget/set_file_image_callbacks() API calls allow an application to hook the memory management operations used when allocating, duplicating, and discarding file images in the property list, in the Core file driver, and potentially in any in-memory file driver developed in the future.

    +

    From the perspective of the HDF5 Library, the supplied image_malloc(), image_memcpy(), image_realloc(), and image_free() callback routines must function identically to the C standard library malloc(), memcpy(), realloc(), and free() calls. What happens on the application side can be much more nuanced, particularly with the ability to pass user data to the callbacks. However, whatever the application does with these calls, it must maintain the illusion that the calls have had the expected effect. Maintaining this illusion requires some understanding of how the property list structure works, and what HDF5 will do with the initial images passed to it.

    +

    At the beginning of this document, we talked about the need to work within the constraints of the property list mechanism. When we said “from the perspective of the HDF5 Library…” in the paragraph above, we are making reference to this point.

    +

    The property list mechanism was developed as a way to add parameters to functions without changing the parameter list and breaking existing code. However, it was designed to use only “call by value” semantics, not “call by reference”. The decision to use “call by value” semantics requires that the values of supplied variables be copied into the property list. This has the advantage of simplifying the copying and deletion of property lists. However, if the value to be copied is large (say a 2 GB file image), the overhead can be unacceptable.

    +

    The usual solution to this problem is to use “call by reference” where only a pointer to an object is placed in a parameter list rather than a copy of the object itself. However, use of “call by reference” semantics would greatly complicate the property list mechanism: at a minimum, it would be necessary to maintain reference counts to dynamically allocated objects so that the owner of the object would know when it was safe to free the object.

    +

    After much discussion, we decided that the file image operations calls were sufficiently specialized that it made no sense to rework the property list mechanism to support “call by reference.” Instead we provided the file image callback mechanism to allow the user to implement some version of “call by reference” when needed. It should be noted that we expect this mechanism to be used rarely if at all. For small file images, the copying overhead should be negligible, and for large images, most use cases should be addressed by the H5LTopen_file_image call.

    +

    In the (hopefully) rare event that use of the file image callbacks is necessary, the fundamental point to remember is that the callbacks must be constructed and used in such a way as to maintain the library’s illusion that it is using “call by value” semantics.

    +

    Thus the property list mechanism must think that it is allocating a new buffer and copying the supplied buffer into it when the file image property is set. Similarly, it must think that it is allocating a new buffer and copying the contents of the existing buffer into it when it copies a property list that contains a file image. Likewise, it must think it is de-allocating a buffer when it discards a property list that contains a file image.

    +

    Similar illusions must be maintained when a file image buffer is copied into the Core file driver (or any future driver that uses the file image callbacks) when the file driver re-sizes the buffer containing the image and finally when the driver discards the buffer.

    + +

    3.1.1. Buffer Ownership

    +

    The owner of a file image in a buffer is the party that has the responsibility to discard the file image buffer when it is no longer needed. In this context, the owner is either the HDF5 Library or the application program.

    +

    We implemented the image_* callback facility to allow efficient management of large file images. These facilities can be used to allow sharing of file image buffers between the application and the HDF5 library, and also transfer of ownership in either direction. In such operations, care must be taken to ensure that ownership is clear and that file image buffers are not discarded before all references to them are discarded by the non-owning party.

    +

    Ownership of a file image buffer will only be passed to the application program if the file image callbacks are designed to do this. In such cases, the application program must refrain from freeing the buffer until the library has deleted all references to it. This in turn will happen after all property lists (if any) that refer to the buffer have been discarded, and the file driver (if any) that used the buffer has closed the file and thinks it has discarded the buffer.

    + +

    3.1.2. Sharing a File image Buffer with the HDF5 Library

    +

    As mentioned above, the HDF5 property lists are a mechanism for passing values into HDF5 Library calls. They were created to allow calls to be extended with new parameters without changing the actual API or breaking existing code. They were designed based on the assumption that all new parameters would be “call by value” and not “call by reference.” Having “call by value” parameters means property lists can be copied, reused, and discarded with ease.

    +

    Suppose an application wished to share a file image buffer with the HDF5 Library. This means the library would be allowed to read the file image, but not free it. The file image callbacks might be constructed as follows to share a buffer:

    + +
      +
    • Construct the image_malloc() call so that it returns the address of the buffer instead of allocating new space. This will keep the library thinking that the buffers are distinct even when they are not. Support this by including the address of the buffer in the user data. As a sanity check, include the buffer’s size in the user data as well, and require image_malloc() to fail if the requested buffer size is unexpected. Finally, include a reference counter in the user data, and increment the reference counter on each call to image_malloc().
      • +
    • Construct the image_memcpy() call so that it does nothing. As a sanity check, make it fail if the source and destination pointers do not match the buffer address in the user data or if the size is unexpected.
      • +
    • Construct the image_free() routine so that it does nothing. As a sanity check, make it compare the supplied pointer with the expected pointer in the user data. Also, make it decrement the reference counter and notify the application that the HDF5 Library is done with the buffer when the reference count drops to 0.
      • +
    + +As the property list code will never resize a buffer, we do not discuss the image_realloc() call here. The behavior of image_realloc() in this scenario depends on what the application wants to do with the file image after it has been opened. We discuss this issue in the next section. Note also that the operation passed into the file image callbacks allow the callbacks to behave differently depending on the context in which they are used.

    +

    For more information on user defined data, see the “H5Pset_file_image_callbacks” section.

    + +

    3.1.3. File Driver Considerations

    +

    When a file image is opened by a driver that sets both the H5FD_FEAT_ALLOW_FILE_IMAGE and the H5FD_FEAT_CAN_USE_FILE_IMAGE_CALLBACKS flags, the driver will allocate a buffer large enough for the initial file image and then copy the image from the property list into this buffer. As processing progresses, the driver will reallocate the image as necessary to increase its size and will eventually discard the image at file close. If defined, the driver will use the file image callbacks for these operations; otherwise, the driver will use the standard C library calls. See the "H5Pset_file_image_callbacks” section for more information.

    +

    As described above, the file image callbacks can be constructed so as to avoid the overhead of buffer allocations and copies while allowing the HDF5 Library to maintain its illusions on the subject. There are two possible complications involving the file driver. The complications are the possibility of reallocation calls from the driver and the possibility of the continued existence of property lists containing references to the buffer.

    +

    Suppose an application wishes to share a file image buffer with the HDF5 Library. The application allows the library to read (and possibly write) the image, but not free it. We must first decide whether the image is to be opened read-only or read/write.

    +

    If the image will be opened read-only (or if we know that any writes will not change the size of the image), the image_realloc() call should never be invoked. Thus the image_realloc() routine can be constructed so as to always fail, and the image_malloc(), image_memcpy(), and image_free() routines can be constructed as described in the section above.

    +

    Suppose, however, that the file image will be opened read/write and may grow during the computation. We must now allow for the base address of the buffer to change due to reallocation calls, and we must employ the user data structure to communicate any change in the buffer base address and size to the application. We pass buffer changes to the application so that the application will be able to eventually free the buffer. To this end, we might define a user data structure as shown in the example below:

    +
     typedef struct udata {
+      void *init_ptr;
+      size_t init_size;
+      int init_ref_count;
+      void *mod_ptr;
+      size_t mod_size;
+      int mod_ref_count;
+ }
+
    +

    Example 1. Using a user data structure to communicate with an application

    +

    We initialize an instance of the structure so that init_ptr points to the buffer to be shared, init_size contains the initial size of the buffer, and all other fields are initialized to either NULL or 0 as indicated by their type. We then pass a pointer to the instance of the user data structure to the HDF5 Library along with allocation callback functions constructed as follows:

    + +
      +
    • Construct the image_malloc() call so that it returns the value in the init_ptr field of the user data structure and increments the init_ref_count. As a sanity check, the function should fail if the requested size does not match the init_size field in the user data structure or if any of the modified fields have values other than their initial values.
      • +
    • Construct the image_memcpy() call so that it does nothing. As a sanity check, it should be made to fail if the source, destination, and size parameters do not match the init_ptr and init_size fields as appropriate.
      • +
    • Construct the image_realloc() call so that it performs a standard realloc. Sanity checking, assuming that the realloc is successful, should be as follows:
      • + +
        +
      • If the mod_ptr, mod_size, or mod_ref_count fields of the user data structure still have their initial values, verify that the supplied pointer matches the init_ptr field and that the supplied size does not match the init_size field. Decrement init_ref_count, set mod_ptr equal to the address returned by realloc, set mod_size equal to the supplied size, and set mod_ref_count to 1.
        • +
      • If the mod_ptr, mod_size, or mod_ref_count fields of the user data structure are defined, verify that the supplied pointer matches the value of mod_ptr and that the supplied size does not match mod_size. Set mod_ptr equal to the value returned by realloc, and set mod_size equal to the supplied size.
        • +
      + +In both cases, if all sanity checks pass, return the value returned by the realloc call. Otherwise, return NULL.

      +
    • Construct the image_free() routine so that it does nothing. Perform sanity checks as follows:
      • +
        +
      • If the H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE flag is set, decrement the init_ref_count field of the user data structure. Flag an error if init_ref_count drops below zero.
        • +
      • If the H5_FILE_IMAGE_OP_FILE_CLOSE flag is set, check to see if the mod_ptr, mod_size, or mod_ref_count fields of the user data structure have been modified from their initial values. If they have, verify that mod_ref_count contains 1 and then set that field to zero. If they have not been modified, proceed as per the H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE case.
        • +
      +
    + +In either case, if both the init_ref_count and mod_ref_count fields have dropped to zero, notify the application that the HDF5 Library is done with the buffer. If the mod_ptr or mod_size fields have been modified, pass these values on to the application as well.

    + +

    3.2. Initial File Image Semantics

    +

    One can argue whether creating a file with an initial file image is closer to creating a file or opening a file. The consensus seems to be that it is closer to a file open, and thus we shall require that the initial image only be used for calls to H5Fopen().

    +

    Whatever our convention, from an internal perspective, opening a file with an initial file image is a bit of both creating a file and opening a file. Conceptually, we will create a file on disk, write the supplied image to the file, close the file, open the file as an HDF5 file, and then proceed as usual (of course, the Core VFD will not write to the file system unless it is configured to do so). This process is similar to a file create: we are creating a file that did not exist on disk to begin with and writing data to it. Also, we must verify that no file of the supplied name is open. However, this process is also similar to a file open: we must read the superblock and handle the usual file open tasks.

    +

    Implementing the above sequence of actions has a number of implications on the behavior of the H5Fopen() call when an initial file image is supplied:

    +
      +
    • H5Fopen() must fail if the target file driver does not set the H5FD_FEAT_ALLOW_FILE_IMAGE flag and a file image is specified in the FAPL.
      • +
    • If the target file driver supports the H5FD_FEAT_ALLOW_FILE_IMAGE flag, then H5Fopen() must fail if the file is already open or if a file of the specified name exists.
      • +
    • Even if the above constraints are satisfied, H5Fopen() must still fail if the image does not contain a valid (or perhaps just plausibly valid) image of an HDF5 file. In particular, the superblock must be processed, and the file structure be set up accordingly.
      • +
    + +See the “Virtual File Driver Feature Flags” section for more information.

    +

    As we indicated earlier, if an initial file image appears in the property list of an H5Fcreate() call, it is ignored.

    +

    While the above section on the semantics of the file image callbacks may seem rather gloomy, we get the payback here. The above says everything that needs to be said about initial file image semantics in general. The sub-section below has a few more observations on the Core file driver.

    + +

    3.2.1. Applying Initial File Image Semantics to the Core File Driver

    +

    At present, the Core file driver uses the open() and read() system calls to load an HDF5 file image from the file system into RAM. Further, if the backing_store flag is set in the FAPL entry specifying the use of the Core file driver, the Core file driver’s internal image will be used to overwrite the source file on either flush or close. See the H5Pset_fapl_core entry in the HDF5 Reference Manual for more information.

    +

    This results in the following observations. In all cases assume that use of the Core file driver has been specified in the FAPL.

    +
      +
    • If the file specified in the H5Fopen() call does not exist, and no initial image is specified in the FAPL, the open must fail because there is no source for the initial image needed by the Core file driver.
      • +
    • If the file specified in the H5Fopen() call does exist, and an initial image is specified in the FAPL, the open must fail because the source of the needed initial image is ambiguous: the file image could be taken either from file or from the FAPL.
      • +
    • If the file specified in the H5Fopen() call does not exist, and an initial image is specified in the FAPL, the open will succeed. This assumes that the supplied image is valid. Further, if the backing store flag is set, the file specified in the H5Fopen() call will be created, and the contents of the Core file driver’s internal buffer will be written to the new file on flush or close.
      • +
    + +Thus a call to H5Fopen() can result in the creation of a new HDF5 file in the file system.

    + +

    4. Examples

    +

    The purpose of this chapter is to provide examples of how to read or build an in-memory HDF5 file image.

    + +

    4.1. Reading an In-memory HDF5 File Image

    + +

    The H5Pset_file_image() function call allows the Core file driver to be initialized from an application provided buffer. The following pseudo code illustrates its use:

    + +
    
+<allocate and initialize buf_len and buf>
+<allocate fapl_id>
+<set fapl to use Core file driver>
+H5Pset_file_image(fapl_id, buf, buf_len);
+<discard buf any time after this point>
+<open file>
+<discard fapl any time after this point>
+<read and/or write file as desired, close>
+
    + +Example 2. Using H5Pset_file_image to initialize the Core file driver

    + +

    This solution is easy to code, but the supplied buffer is duplicated twice. The first time is in the call to H5Pset_file_image() when the image is duplicated and the duplicate inserted into the property list. The second time is when the file is opened: the image is copied from the property list into the initial buffer allocated by the Core file driver. This is a non-issue for small images, but this could become a significant performance hit for large images.

    +

    If we want to avoid the extra malloc and memcpycalls, we must decide whether the application should retain ownership of the buffer or pass ownership to the HDF5 Library.

    +

    The following pseudo code illustrates opening the image read -only using the H5LTopen_file_image() routine. In this example, the application retains ownership of the buffer and avoids extra buffer allocations and memcpy calls.

    + +
    
+<allocate and initialize buf_len and buf>
+hid_t file_id;
+unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
+file_id = H5LTopen_file_image(buf, buf_len, flags);
+<read file as desired, and then close>
+<discard buf any time after this point>
+
    + +Example 3. Using H5LTopen_file_image to open a read-only file image where the application retains ownership of the buffer + +

    If the application wants to transfer ownership of the buffer to the HDF5 Library, and the standard C library routine free is an acceptable way of discarding it, the above example can be modified as follows:

    + +
    
+<allocate and initialize buf_len and buf>
+hid_t file_id;
+unsigned flags = H5LT_FILE_IMAGE_DONT_COPY;
+file_id = H5LTopen_file_image(buf, buf_len, flags);
+<read file as desired, and then close>
+
    + +Example 4. Using H5LTopen_file_image to open a read-only file image where the application transfers ownership of the buffer + +

    Again, file access is read-only. Read/write access can be obtained via the H5LTopen_file_image() call, but we will explore that in the section below.

    + +

    4.2. In-memory HDF5 File Image Construction

    + +

    Before the implementation of file image operations, HDF5 supported construction of an image of an HDF5 file in memory with the Core file driver. The H5Fget_file_image() function call allows an application access to the file image without first writing it to disk. See the following code fragment:

    + +
    
+<Open and construct the desired file with the Core file driver>
+H5Fflush(fid);
+size = H5Fget_file_image(fid, NULL, 0);
+buffer_ptr = malloc(size);
+H5Fget_file_image(fid, buffer_ptr, size);
+
    + +Example 5. Accessing the image of a file in memory + +

    The use of H5Fget_file_image() may be acceptable for small images. For large images, the cost of the malloc() and memcpy() operations may be excessive. To address this issue, the H5Pset_file_image_callbacks() call allows an application to manage dynamic memory allocation for file images and memory-based file drivers (only the Core file driver at present). The following code fragment illustrates its use. Note that most error checking is omitted for simplicity and that H5Pset_file_image is not used to set the initial file image.

    + +
    
+    struct udata_t {
+    void * image_ptr;
+    size_t image_size;
+          } udata = {NULL, 0};
+
+void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    ((struct udata_t *)udata)->image_size = size;
+    return(malloc(size));
+}
+
+void *image_memcpy)(void *dest, const void *src, size_t size,
+ H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(FALSE); /* Should never be invoked in this scenario. */
+    return(NULL); /* always fails */
+}
+
+void image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    ((struct udata_t *)udata)->image_size = size;
+    return(realloc(ptr, size));
+}
+
+herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE);
+    ((struct udata_t *)udata)->image_ptr = ptr;
+    return(0); /* if we get here, we must have been successful */
+}
+
+void *udata_copy(void *udata)
+{
+    return(udata);
+}
+
+herr_t udata_free(void *udata)
+{
+    return(0);
+}
+
+H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
+                                       image_realloc, image_free,
+                                       udata_copy, udata_free,
+                                       (void *)(&udata)};
+
+<allocate fapl_id>
+H5Pset_file_image_callbacks(fapl_id, &callbacks);
+<open core file using fapl_id, write file, close it>
+assert(udata.image_ptr!= NULL);
+/* udata now contains the base address and length of the final version of the core file */
+<use image of file, and then discard it via free()>
+
    + +Example 6. Using H5Pset_file_image_callbacks to improve memory allocation

    + +

    The above code fragment gives the application full ownership of the buffer used by the Core file driver after the file is closed, and it notifies the application that the HDF5 Library is done with the buffer by setting udata.image_ptr to something other than NULL. If read access to the buffer is sufficient, the H5Fget_vfd_handle() call can be used as an alternate solution to get access to the base address of the Core file driver’s buffer.

    +

    The above solution avoids some unnecessary mallocand memcpycalls and should be quite adequate if an image of an HDF5 file is constructed only occasionally. However, if an HDF5 file image must be constructed regularly, and if we can put a strong and tight upper bound on the size of the necessary buffer, then the following pseudo code demonstrates a method of avoiding memory allocation completely. The downside, however, is that buffer is allocated statically. Again, much error checking is omitted for clarity.

    + +
    
+char buf[BIG_ENOUGH];
+struct udata_t {
+void * image_ptr;
+size_t image_size;
+size_t max_image_size;
+int ref_count;
+} udata = {(void *)(&(buf[0]), 0, BIG_ENOUGH, 0};
+
+void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(size <= ((struct udata_t *)udata)->max_image_size);
+    assert(((struct udata_t *)udata)->ref_count == 0);
+    ((struct udata_t *)udata)->image_size = size;
+    (((struct udata_t *)udata)->ref_count)++;
+    return((((struct udata_t *)udata)->image_ptr);
+}
+void *image_memcpy)(void *dest, const void *src, size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(FALSE); /* Should never be invoked in this scenario. */
+    return(NULL); /* always fails */
+}
+void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(ptr == ((struct udata_t *)udata)->image_ptr);
+    assert(size <= ((struct udata_t *)udata)->max_image_size);
+    assert(((struct udata_t *)udata)->ref_count == 1);
+    ((struct udata_t *)udata)->image_size = size;
+    return((((struct udata_t *)udata)->image_ptr);
+}
+herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE);
+    assert(ptr == ((struct udata_t *)udata)->image_ptr);
+    assert(((struct udata_t *)udata)->ref_count == 1);
+    (((struct udata_t *)udata)->ref_count)--;
+    return(0); /* if we get here, we must have been successful */
+}
+void *udata_copy(void *udata)
+{
+    return(udata);
+}
+herr_t udata_free(void *udata)
+{
+    return(0);
+}
+H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
+                                       image_realloc, image_free,
+                                       udata_copy, udata_free,
+                                       (void *)(&udata)};
+/* end of initialization */
+<allocate fapl_id>
+H5Pset_file_image_callbacks(fapl_id, &callbacks);
+<open core file using fapl_id>
+<discard fapl any time after the open>
+<write the file, flush it, and then close it>
+assert(udata.ref_count == 0);
+/* udata now contains the base address and length of the final version of the core file */
+<use the image of the file>
+<reinitialize udata, and repeat the above from the end of initialization onwards to write a new file image>
+
    + +Example 7. Using H5Pset_file_image_callbacks with a static buffer

    + +

    If we can further arrange matters so that only the contents of the datasets in the HDF5 file image change, but not the structure of the file itself, we can optimize still further by re-using the image and changing only the contents of the datasets after the initial write to the buffer. The following pseudo code shows how this might be done. Note that the code assumes that buf already contains the image of the HDF5 file whose dataset contents are to be overwritten. Again, much error checking is omitted for clarity. Also, observe that the file image callbacks do not support the H5Pget_file_image() call.

    + +
    
+<buf already defined and loaded with file image>
+<udata already defined and initialized>
+void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(size <= ((struct udata_t *)udata)->max_image_size);
+    assert(size == ((struct udata_t *)udata)->image_size);
+    assert(((struct udata_t *)udata)->ref_count >= 0);
+          ((struct udata_t *)udata)->image_size = size;
+          (((struct udata_t *)udata)->ref_count)++;
+    return((((struct udata_t *)udata)->image_ptr);
+}
+void *image_memcpy)(void *dest, const void *src, size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(dest == ((struct udata_t *)udata)->image_ptr);
+    assert(src == ((struct udata_t *)udata)->image_ptr);
+    assert(size <= ((struct udata_t *)udata)->max_image_size);
+    assert(size == ((struct udata_t *)udata)->image_size);
+    assert(((struct udata_t *)udata)->ref_count >= 1);
+    return(dest); /* if we get here, we must have been successful */
+}
+void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    /* One would think that this function is not needed in this scenario, as
+     *  only the contents of the HDF5 file is being changed, not its size or
+     *  structure. However, the Core file driver calls realloc() just before
+     *  close to clip the buffer to the size indicated by the end of the
+     *  address space.
+     *
+     *  While this call must be supported in this case, the size of
+     *  the image should never change. Hence the function can limit itself
+     *  to performing sanity checks, and returning the base address of the
+     *  statically allocated buffer.
+     */
+    assert(ptr == ((struct udata_t *)udata)->image_ptr);
+    assert(size <= ((struct udata_t *)udata)->max_image_size);
+    assert(((struct udata_t *)udata)->ref_count >= 1);
+    assert(((struct udata_t *)udata)->image_size == size);
+    return((((struct udata_t *)udata)->image_ptr);
+}
+herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert((file_image_op == H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE) ||
+           (file_image_op == H5_FILE_IMAGE_OP_FILE_CLOSE));
+    assert(((struct udata_t *)udata)->ref_count >= 1);
+          (((struct udata_t *)udata)->ref_count)--;
+    return(0); /* if we get here, we must have been successful */
+}
+void *udata_copy(void *udata)
+{
+    return(udata);
+}
+herr_t udata_free(void *udata)
+{
+    return(0);
+}
+H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
+                                       image_realloc, image_free,
+                                       udata_copy, udata_free,
+                                       (void *)(&udata)};
+/* end of initialization */
+<allocate fapl_id>
+H5Pset_file_image_callbacks(fapl_id, &callbacks);
+H5Pset_file_image(fapl_id, udata.image_ptr, udata.image_len);
+<open core file using fapl_id>
+<discard fapl any time after the open>
+<overwrite data in datasets in the file, and then close it>
+assert(udata.ref_count == 0);
+/* udata now contains the base address and length of the final version of the core file */
+<use the image of the file>
+<repeat the above from the end of initialization onwards to write new data to datasets in file image>
+
    + +Example 8. Using H5Pset_file_image_callbacks where only the datasets change + +

    Before we go on, we should note that the above pseudo code can be written more compactly, albeit with fewer sanity checks, using the H5LTopen_file_image() call. See the example below:

    + +
    
+<buf already defined and loaded with file image>
+<udata already defined and initialized>
+hid_t file_id;
+unsigned flags = H5LT_FILE_IMAGE_OPEN_RW | H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
+/* end initialization */
+file_id = H5LTopen_file_image(udata.image_ptr, udata.image_len, flags);
+<overwrite data in datasets in the file, and then close it>
+/* udata now contains the base address and length of the final version of the core file */
+<use the image of the file>
+<repeat the above from the end of initialization onwards to write new data to datasets in file image>
+
    + +

    Example 9. Using H5LTopen_file_image where only the datasets change +

    The above pseudo code allows updates of a file image about as cheaply as possible. We assume the application has enough RAM for the image and that the HDF5 file structure is constant after the first write.

    +

    While the scenario above is plausible, we will finish this section with a more general scenario. In the pseudo code below, we assume sufficient RAM to retain the HDF5 file image between uses, but we do not assume that the HDF5 file structure remains constant or that we can place a hard pper bound on the image size.

    +

    Since we must use malloc, realloc, and free in this example, and since realloc can change the base address of a buffer, we must maintain two of ptr, size, and ref_count triples in the udata structure. The first triple is for the property list (which will never change the buffer), and the second triple is for the file driver. As shall be seen, this complicates the file image callbacks considerably. Note also that while we do not use H5Pget_file_image() in this example, we do include support for it in the file image callbacks. As usual, much error checking is omitted in favor of clarity.

    + +
    
+struct udata_t {
+    void * fapl_image_ptr;
+    size_t fapl_image_size;
+    int fapl_ref_count;
+    void * vfd_image_ptr;
+    size_t vfd_image_size;
+    nt vfd_ref_count;
+    } udata = {NULL, 0, 0, NULL, 0, 0};
+boolean initial_file_open = TRUE;
+
+void *image_malloc(size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    void * return_value = NULL;
+    switch ( file_image_op ) {
+        case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET:
+        case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY:
+            assert(((struct udata_t *)udata)->fapl_image_ptr != NULL);
+            assert(((struct udata_t *)udata)->fapl_image_size == size);
+            assert(((struct udata_t *)udata)->fapl_ref_count >= 0);
+            return_value = ((struct udata_t *)udata)->fapl_image_ptr;
+            (((struct udata_t *)udata)->fapl_ref_count)++;
+            break;
+        case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET:
+            assert(((struct udata_t *)udata)->fapl_image_ptr != NULL);
+            assert(((struct udata_t *)udata)->vfd_image_size == size);
+            assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
+            return_value = ((struct udata_t *)udata)->fapl_image_ptr;
+            /* don’t increment ref count */
+            break;
+        case H5_FILE_IMAGE_OP_FILE_OPEN:
+            assert(((struct udata_t *)udata)->vfd_image_ptr == NULL);
+            assert(((struct udata_t *)udata)->vfd_image_size == 0);
+            assert(((struct udata_t *)udata)->vfd_ref_count == 0);
+            if (((struct udata_t *)udata)->fapl_image_ptr == NULL ) {
+                ((struct udata_t *)udata)->vfd_image_ptr = malloc(size);
+                ((struct udata_t *)udata)->vfd_image_size = size;
+            } else {
+                assert(((struct udata_t *)udata)->fapl_image_size == size);
+                assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
+                ((struct udata_t *)udata)->vfd_image_ptr = ((struct udata_t *)udata)->fapl_image_ptr;
+                ((struct udata_t *)udata)->vfd_image_size = size;
+            }
+            return_value = ((struct udata_t *)udata)->vfd_image_ptr;
+            (((struct udata_t *)udata)->vfd_ref_count)++;
+            break;
+        default:
+            assert(FALSE);
+          }
+    return(return_value);
+}
+
+void *image_memcpy)(void *dest, const void *src, size_t size,
+ H5_file_image_op_t file_image_op, void *udata)
+{
+    switch(file_image_op) {
+        case H5_FILE_IMAGE_OP_PROPERTY_LIST_SET:
+        case H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY:
+        case H5_FILE_IMAGE_OP_PROPERTY_LIST_GET:
+            assert(dest == ((struct udata_t *)udata)->fapl_image_ptr);
+            assert(src == ((struct udata_t *)udata)->fapl_image_ptr);
+            assert(size == ((struct udata_t *)udata)->fapl_image_size);
+            assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
+            break;
+        case H5_FILE_IMAGE_OP_FILE_OPEN:
+            assert(dest == ((struct udata_t *)udata)->vfd_image_ptr);
+            assert(src == ((struct udata_t *)udata)->fapl_image_ptr);
+            assert(size == ((struct udata_t *)udata)->fapl_image_size);
+            assert(size == ((struct udata_t *)udata)->vfd_image_size);
+            assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
+            assert(((struct udata_t *)udata)->vfd_ref_count == 1);
+            break;
+        default:
+            assert(FALSE);
+            break;
+        }
+    return(dest); /* if we get here, we must have been successful */
+}
+
+void *image_realloc(void *ptr, size_t size, H5_file_image_op_t file_image_op, void *udata)
+{
+    assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr); |
+    assert(((struct udata_t *)udata)->vfd_ref_count == 1);
+    ((struct udata_t *)udata)->vfd_image_ptr = realloc(ptr, size);
+    ((struct udata_t *)udata)->vfd_image_size = size;
+    return((((struct udata_t *)udata)->vfd_image_ptr);
+}
+
+herr_t image_free(void *ptr, H5_file_image_op_t file_image_op, void *udata)
+{
+    switch(file_image_op) {
+        case H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE:
+            assert(ptr == ((struct udata_t *)udata)->fapl_image_ptr);
+            assert(((struct udata_t *)udata)->fapl_ref_count >= 1);
+            (((struct udata_t *)udata)->fapl_ref_count)--;
+            break;
+        case H5_FILE_IMAGE_OP_FILE_CLOSE:
+            assert(ptr == ((struct udata_t *)udata)->vfd_image_ptr);
+            assert(((struct udata_t *)udata)->vfd_ref_count == 1);
+            (((struct udata_t *)udata)->vfd_ref_count)--;
+            break;
+        default:
+            assert(FALSE);
+            break;
+        }
+    return(0); /* if we get here, we must have been successful */
+}
+
+void *udata_copy(void *udata)
+{
+          return(udata);
+}
+
+herr_t udata_free(void *udata)
+{
+          return(0);
+}
+H5_file_image_callbacks_t callbacks = {image_malloc, image_memcpy,
+                                       image_realloc, image_free,
+                                       udata_copy, udata_free,
+                                       (void *)(&udata)};
+/* end of initialization */
+<allocate fapl_id>
+H5Pset_file_image_callbacks(fapl_id, &callbacks);
+if ( initial_file_open ) {
+    initial_file_open = FALSE;
+} else {
+    assert(udata.vfd_image_ptr != NULL);
+    assert(udata.vfd_image_size > 0);
+    assert(udata.vfd_ref_count == 0);
+    assert(udata.fapl_ref_count == 0);
+    udata.fapl_image_ptr = udata.vfd_image_ptr;
+    udata.fapl_image_size = udata.vfd_image_size;
+    udata.vfd_image_ptr = NULL;
+    udata.vfd_image_size = 0;
+    H5Pset_file_image(fapl_id, udata.fapl_image_ptr, udata.fapl_image_size);
+}
+
+<open core file using fapl_id>
+<discard fapl any time after the open>
+<write/update the file, and then close it>
+assert(udata.fapl_ref_count == 0);
+assert(udata.vfd_ref_count == 0);
    
+
    /* udata.vfd_image_ptr and udata.vfd_image_size now contain the base address and length of the final version of the core file */
+<use the image of the file>
+<repeat the above from the end of initialization to modify the file image as needed>
+<free the image when done>
+
    + +Example 10. Using H5LTopen_file_image where only the datasets change and where the file structure and image size might not be constant

    + +

    The above pseudo code shows how a buffer can be passed back and forth between the application and the HDF5 Library. The code also shows the application having control of the actual allocation, reallocation, and freeing of the buffer.

    + +

    4.3. Using HDF5 to Construct and Read a Data Packet

    +

    Using the file image operations described in this document, we can bundle up data in an image of an HDF5 file on one process, transmit the image to a second process, and then open and read the image on the second process without any mandatory file system I/O.

    +

    We have already demonstrated the construction and reading of such buffers above, but it may be useful to offer an example of the full operation. We do so in the example below using as simple a set of calls as possible. The set of calls in the example has extra buffer allocations. To reduce extra buffer allocations, see the sections above.

    +

    In the following example, we construct an HDF5 file image on process A and then transmit the image to process B where we then open the image and extract the desired data. Note that no file system I/O is performed: all the processing is done in memory with the Core file driver.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    *** Process A ****** Process B ***
    <Open and construct the desired file with the Core file driver>hid_t file_id;

    H5Fflush(fid);
    size = H5Fget_file_image(fid, NULL, 0);
    buffer_ptr = malloc(size);
    H5Fget_file_image(fid, buffer_ptr, size);
    <transmit size>
    <receive size>
    <transmit *buffer_ptr>buffer_ptr = malloc(size)
    free(buffer_ptr);<receive image in *buffer_ptr>
    <close core file>file_id = H5LTopen_file_image(buf, + buf_len, + H5LT_FILE_IMAGE_DONT_COPY);
    <read data from file, then close. note that the Core file driver will discard the buffer on close>
    + +Example 11. Building and passing a file image from one process to another + + +

    4.4. Using a Template File

    + +

    After the above examples, an example of the use of a template file might seem anti-climactic. A template file might be used to enforce consistency on file structure between files or in parallel HDF5 to avoid long sequences of collective operations to create the desired groups, datatypes, and possibly datasets. The following pseudo code outlines a potential use:

    + +
    
+<allocate and initialize buf and buflen, with buf containing the desired initial image (which in turn contains the desired group, datatype, and dataset definitions), and buf_len containing the size of buf>
+<allocate fapl_id>
+<set fapl to use desired file driver that supports initial images>
+H5Pset_file_image(fapl_id, buf, buf_len);
+<discard buf any time after this point>
+<open file>
+<discard fapl any time after this point>
+<read and/or write file as desired, close>
+
    + +Example 12. Using a template file

    + +

    Observe that the above pseudo code includes an unnecessary buffer allocation and copy in the call to H5Pset_file_image(). As we have already discussed ways of avoiding this, we will not address that issue here.

    +

    What is interesting in this case is to consider why the application would find this use case attractive.

    +

    In the serial case, at first glance there seems little reason to use the initial image facility at all. It is easy enough to use standard C calls to duplicate a template file, rename it as desired, and then open it as an HDF5 file.

    +

    However, this assumes that the template file will always be available and in the expected place. This is a questionable assumption for an application that will be widely distributed. Thus, we can at least make an argument for either keeping an image of the template file in the executable or for including code for writing the desired standard definitions to new HDF5 files.

    +

    Assuming the image is relatively small, we can further make an argument for the image in place of the code, as, quite simply, the image should be easier to maintain and modify with an HDF5 file editor.

    +

    However, there remains the question of why one should pass the image to the HDF5 Library instead of writing it directly with standard C calls and then using HDF5 to open it. Other than convenience and a slight reduction in code size, we are hard pressed to offer a reason.

    +

    In contrast, the argument is stronger in the parallel case since group, datatype, and dataset creations are all expensive collective operations. The argument is also weaker: simply copying an existing template file and opening it should lose many of its disadvantages in the HPC context although we would imagine that it is always useful to reduce the number of files in a deployment.

    +

    In closing, we would like to consider one last point. In the parallel case, we would expect template files to be quite large. Parallel HDF5 requires eager space allocation for chunked datasets. For similar reasons, we would expect template files in this context to contain long sequences of zeros with a scattering of metadata here and there. Such files would compress well, and the compressed images would be cheap to distribute across the available processes if necessary. Once distributed, each process could uncompress the image and write to file those sections containing actual data that lay within the section of the file assigned to the process. This approach might be significantly faster than a simple copy as it would allow sparse writes, and thus it might provide a compelling use case for template files. However, this approach would require extending our current API to allow compressed images. We would also have to add the H5Pget/set_image_decompression_callback() API calls. We see no problem in doing this. However, it is beyond the scope of the current effort, and thus we will not pursue the matter further unless there is interest in our doing so.

    + +

    5. Java Signatures for File Image Operations API Calls

    + +

    Potential Java function call signatures for the file image operation APIs are described in this section. These have not yet been implemented, and there are no immediate plans for implementation.

    +

    Note that the H5LTopen_file_image() call is omitted. We have not supported high-level library calls in Java.

    + +

    H5Pset_file_image

    +

    int H5Pset_file_image(int fapl_id, const byte[] buf_ptr);

    + +H5Pget_file_image

    +

    herr_t H5Pget_file_image(hid_t fapl_id, byte[] buf_ptr_ptr);

    + +H5_file_image_op_t

    +
    public static H5_file_image_op_t
+{
+     H5_FILE_IMAGE_OP_PROPERTY_LIST_SET,
+     H5_FILE_IMAGE_OP_PROPERTY_LIST_COPY,
+     H5_FILE_IMAGE_OP_PROPERTY_LIST_GET,
+     H5_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE,
+     H5_FILE_IMAGE_OP_FILE_OPEN,
+     H5_FILE_IMAGE_OP_FILE_RESIZE,
+     H5_FILE_IMAGE_OP_FILE_CLOSE
+}
    + +H5_file_image_malloc_cb

    +
    public interface H5_file_image_malloc_cb extends Callbacks {
+     buf[] callback(H5_file_image_op_t file_image_op, CBuserdata udata);
+}
    + +H5_file_image_memcpy_cb

    +
    public interface H5_file_image_memcpy_cb extends Callbacks {
+buf[] callback(buf[] dest, const buf[] src, H5_file_image_op_t file_image_op, CBuserdata
+udata);
+}
    + +H5_file_image_realloc_cb

    +
    public interface H5_file_image_realloc_cb extends Callbacks {
+     buf[] callback(buf[] ptr, H5_file_image_op_t file_image_op, CBuserdata udata);
+}
    + +H5_file_image_free_cb

    +
    public interface H5_file_image_free_cb extends Callbacks {
+     void callback(buf[] ptr, H5_file_image_op_t file_image_op, CBuserdata udata);
+}
    + +H5_file_udata_copy_cb

    +
    public interface H5_file_udata_copy_cb extends Callbacks {
+     buf[] callback(CBuserdata udata);
+}
    + +H5_file_udata_free_cb

    +
    public interface H5_file_udata_free_cb extends Callbacks {
+     void callback(CBuserdata udata);
+}
    + +H5_file_image_callbacks_t

    +
    public abstract class H5_file_image_callbacks_t
+{
+     H5_file_image_malloc_cb image_malloc;
+     H5_file_image_memcpy_cb image_memcpy;
+     H5_file_image_realloc_cb image_realloc;
+     H5_file_image_free_cb image_free;
+     H5_file_udata_copy_cb udata_copy;
+     H5_file_udata_free_cb udata_free;
+     CBuserdata udata;
+     public H5_file_image_callbacks_t(
+          H5_file_image_malloc_cb image_malloc,
+          H5_file_image_memcpy_cb image_memcpy,
+          H5_file_image_realloc_cb image_realloc,
+          H5_file_image_free_cb image_free,
+          H5_file_udata_copy_cb udata_copy,
+          H5_file_udata_free_cb udata_free,
+          CBuserdata udata) {
+       this.image_malloc = image_malloc;
+       this.image_memcpy = image_memcpy;
+       this.image_realloc = image_realloc;
+       this.image_free = image_free;
+       this.udata_copy = udata_copy;
+       this.udata_free = udata_free;
+       this.udata = udata;
+     }
+}
    + +H5Pset_file_image_callbacks

    +
    int H5Pset_file_image_callbacks(int fapl_id, H5_file_image_callbacks_t callbacks_ptr);
    + +H5Pget_file_image_callbacks

    +
    int H5Pget_file_image_callbacks(int fapl_id, H5_file_image_callbacks_t[] callbacks_ptr);
    + +H5Fget_file_image

    +
    long H5Fget_file_image(int file_id, byte[] buf_ptr);
    + +

    6. Fortran Signatures for File Image Operations API Calls

    + +

    Potential Fortran function call signatures for the file image operation APIs are described in this section. These have not yet been implemented, and there are no immediate plans for implementation.

    + +

    6.1. Low-level Fortran API Routines

    + +

    The Fortran low-level APIs make use of Fortran 2003’s ISO_C_BINDING module in order to achieve portable and standard conforming interoperability with the C APIs. The C pointer (C_PTR) and function pointer (C_FUN_PTR) types are returned from the intrinsic procedures C_LOC(X) and C_FUNLOC(X), respectively, defined in the ISO_C_BINDING module. The argument X is the data or function to which the C pointers point to and must have the TARGET attribute in the calling program. Note that the variable name lengths of the Fortran equivalent of the predefined C constants were shortened to less than 31 characters in order to be Fortran standard compliant.

    + +

    6.1.1. H5Pset_file_image_f

    + +

    The signature of H5Pset_file_image_f is defined as follows:

    +

    SUBROUTINE H5Pset_file_image_f(fapl_id, buf_ptr, buf_len, hdferr) + +

    The parameters of H5Pset_file_image are defined as follows:

    + + + + + + + + + + + + + + + + + + +
    INTEGER(hid_t), INTENT(IN):: fapl_idWill contain the ID of the target file access property list.
    TYPE(C_PTR), INTENT(IN):: buf_ptrWill supply the C pointer to the initial file image or C_NULL_PTR if no initial file image is desired.
    INTEGER(size_t), INTENT(IN):: buf_lenWill contain the size of the supplied buffer or 0 if no initial image is desired.
    INTEGER, INTENT(OUT) :: hdferrWill return the error status: 0 for success and -1 for failure.
    + +

    6.1.2. H5Pget_file_image_f

    +

    The signature of H5Pget_file_image_f is defined as follows:

    +

    SUBROUTINE H5Pget_file_image_f(fapl_id, buf_ptr, buf_len, hdferr) + +

    The parameters of H5Pget_file_image_f are defined as follows:

    + + + + + + + + + + + + + + + + + +
    INTEGER(hid_t), INTENT(IN) :: fapl_idWill contain the ID of the target file access property list
    TYPE(C_PTR), INTENT(INOUT), VALUE :: buf_ptrWill hold either a C_NULL_PTR or a scalar of type c_ptr. If buf_ptr is not C_NULL_PTR, on successful return, buf_ptr shall contain a C pointer to a copy of the initial image provided in the last call to H5Pset_file_image_f for the supplied fapl_id, or buf_ptr shall contain a C_NULL_PTR if there is no initial image set. The Fortran pointer can be obtained using the intrinsic C_F_POINTER.
    INTEGER(size_t), INTENT(OUT) :: buf_lenWill contain the value of the buffer parameter for the initial image in the supplied fapl_id. The value will be 0 if no initial image is set.
    INTEGER, INTENT(OUT) :: hdferrWill return the error status: 0 for success and -1 for failure.
    + +

    6.1.3. H5Pset_file_image_callbacks_f

    +

    The signature of H5Pset_file_image_callbacks_f is defined as follows: + +

    
+
    INTEGER :: H5_IMAGE_OP_PROPERTY_LIST_SET_F=0,
+     H5_IMAGE_OP_PROPERTY_LIST_COPY_F=1,
+     H5_IMAGE_OP_PROPERTY_LIST_GET_F=2,
+     H5_IMAGE_OP_PROPERTY_LIST_CLOSE_F=3,
+     H5_IMAGE_OP_FILE_OPEN_F=4,
+     H5_IMAGE_OP_FILE_RESIZE_F=5,
+     H5_IMAGE_OP_FILE_CLOSE_F=6
+TYPE, BIND(C) :: H5_file_image_callbacks_t
+     TYPE(C_FUN_PTR), VALUE :: image_malloc
+     TYPE(C_FUN_PTR), VALUE :: image_memcpy
+     TYPE(C_FUN_PTR), VALUE :: image_realloc
+     TYPE(C_FUN_PTR), VALUE :: image_free
+     TYPE(C_FUN_PTR), VALUE :: udata
+     TYPE(C_FUN_PTR), VALUE :: udata_copy
+     TYPE(C_FUN_PTR), VALUE :: udata_free
+     TYPE(C_PTR), VALUE :: udata
+END TYPE H5_file_image_callbacks_t
+
    + +The semantics of the above values will be the same as those defined in the C enum. See Section 2.1.3 for more information.

    +

    Fortran Callback APIs

    +

    The Fortran callback APIs are shown below.

    +

    FUNCTION op_func(size, file_image_op, udata,) RESULT(image_malloc) + + + + + + + + + + + + + + + + + + +
    INTEGER(size_t) :: sizeWill contain the size of the image buffer to allocate in bytes.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: image_mallocShall contain a pointer to a function with functionality identical to the standard C library memcpy() call.

    +

    FUNCTION op_func(dest, src, size, & file_image_op, udata) RESULT(image_memcpy) + + + + + + + + + + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), VALUE :: destWill contain the address of the buffer into which to copy.
    TYPE(C_PTR), VALUE :: srcWill contain the address of the buffer from which to copy
    INTEGER(size_t) :: sizeWill contain the number of bytes to copy.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: image_memcpyShall contain a pointer to a function with functionality identical to the standard C library memcpy() call.

    + +

    FUNCTION op_func(ptr, size, & file_image_op, udata) RESULT(image_realloc) + + + + + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), VALUE :: ptrWill contain the pointer to the buffer being reallocated
    INTEGER(size_t) :: sizeWill contain the desired size of the buffer after realloc in bytes.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: image_reallocShall contain a pointer to a unction functionality identical to the standard C library realloc() call.

    + +

    FUNCTION op_func(ptr, file_image_op, udata) RESULT(image_free) + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), VALUE :: ptrWill contain the pointer to the buffer being released.
    INTEGER :: file_image_opWill be set to one of the values of H5_IMAGE_OP_* indicating the operation being performed on the file image when this callback is invoked.
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_PTR), VALUE :: image_freeShall contain a pointer to a function with functionality identical to the standard C library free() call

    + +

    FUNCTION op_func(udata) RESULT(udata_copy) + + + + + + + + + +
    TYPE(C_PTR), VALUE :: udataWill be set to the value passed in for the udata parameter to H5Pset_file_image_callbacks_f.
    TYPE(C_FUN_PTR), VALUE :: udata_copyShall contain a pointer to a function that will allocate a buffer of suitable size, copy the contents of the supplied udata into the new buffer, and return the address of the new buffer. The function will return C_NULL_PTR on failure.

    + +

    FUNCTION op_func(udata) RESULT(udata_free) + + + + + +
    TYPE(C_PTR), VALUE :: udataShall contain a pointer value, potentially to user-defined data, that will be passed to the image_malloc, image_memcpy, image_realloc, and image_free callbacks.

    + +

    The signature of H5Pset_file_image_callbacks_f is defined as follows:

    +

    SUBROUTINE H5Pset_file_image_callbacks_f(fapl_id, &callbacks_ptr, hdferr) +The parameters are defined as follows:

    + + + + + + + + + + + + + +

    INTEGER(hid_t), INTENT(IN) :: fapl_id

    		Will contain the ID of the target file access property list.

    TYPE(H5_file_image_callbacks_t), INTENT(IN) :: callbacks_ptr

    		Will contain the callback derived type. callbacks_ptr shall contain a pointer to the Fortran function via the intrinsic functions C_LOC(X) and C_FUNLOC(X).

    INTEGER, INTENT(OUT) :: hdferr

    		Will return the error status: 0 for success and -1 for failure.

    + +

    6.1.4. H5Pget_file_image_callbacks_f

    +

    The H5Pget_file_image_callbacks_f routine is designed to obtain the current file image callbacks from a file access property list.

    +

    The signature is defined as follows

    +

    SUBROUTINE H5Pget_file_image_callbacks_f(fapl_id, callbacks_ptr, hdferr) +The parameters are defined as follows:

    + + + + + + + + + + + + + +
    INTEGER(hid_t), INTENT(IN) :: fapl_idWill contain the ID of the target file access property list.

    TYPE(H5_file_image_callbacks_t), INTENT(OUT) :: callbacks_ptrWill contain the callback derived type. Each member of the derived type shall have the same meaning as its C counterpart. See section 2.1.4 for more information.

    INTEGER, INTENT(OUT) :: hdferrWill return the error status: 0 for success and -1 for failure.

    + +

    6.1.5. Fortran Virtual File Driver Feature Flags

    +

    Implementation of the H5Pget/set_file_image_callbacks_f() and H5Pget/set_file_image_f() APIs requires a pair of new virtual file driver feature flags:

    +

    H5FD_FEAT_LET_IMAGE_F +H5FD_FEAT_LET_IMAGE_CALLBACK_F +See the “Virtual File Driver Feature Flags” section for more information.

    + +

    6.1.6. H5Fget_file_image_f

    +

    The signature of H5Fget_file_image_f shall be defined as follows:

    +

    SUBROUTINE H5Fget_file_image_f(file_id, buf_ptr, buf_len, hdferr, buf_size) +The parameters of H5Fget_file_image_f are defined as follows:

    + + + + + + + + + + + + + + + + + + + + +

    See the “H5Fget_file_image” section for more information.

    + +
    INTEGER(hid_t), INTENT(IN) :: file_idWill contain the ID of the target file.

    TYPE(C_PTR), INTENT(IN) :: buf_ptrWill contain a C pointer to the buffer into which the image of the HDF5 file is to be copied. If buf_ptr is C_NULL_PTR, no data will be copied.

    INTEGER(size_t), INTENT(IN) :: buf_lenWill contain the size in bytes of the supplied buffer.

    INTEGER(ssizet_t), INTENT(OUT), OPTIONAL :: buf_sizeWill indicate the buffer size required to store the file image (in other words, the length of the file). If only the buf_size is needed, then buf_ptr should be also be set to C_NULL_PTR

    INTEGER, INTENT(OUT) :: hdferrReturns the error status: 0 for success and -1 for failure.

    + +

    6.2. High-level Fortran API Routine

    + +The new Fortran high-level routine H5LTopen_file_image_f will provide a wrapper for the high-level H5LTopen_file_image function. Consequently, the high-level Fortran API will not be implemented using low-level HDF5 Fortran APIs. + +

    6.2.1. H5LTopen_file_image_f

    +The signature of H5LTopen_file_image_f is defined as follows:

    +

    SUBROUTINE H5LTopen_file_image_f(buf_ptr, buf_len, flags, file_id, hdferr) +The parameters of H5LTopen_file_image_f are defined as follows:

    + + + + + + + + + + + + + + + + + + + + + +
    TYPE(C_PTR), INTENT(IN), VALUE :: buf_ptrWill contain a pointer to the supplied initial image. A C_NULL_PTR value is invalid and will cause H5LTopen_file_image_f to fail.
    INTEGER(size_t), INTENT(IN) :: buf_lenWill contain the size of the supplied buffer. A value of 0 is invalid and will cause H5LTopen_file_image_f to fail.
    INTEGER, INTENT(IN) :: flagsWill contain a set of flags indicating whether the image is to be opened read/write, whether HDF5 is to take control of the buffer, and how long the application promises to maintain the buffer. Possible flags are as follows: H5LT_IMAGE_OPEN_RW_F, H5LT_IMAGE_DONT_COPY_F, and H5LT_IMAGE_DONT_RELEASE_F. The C equivalent flags are defined in the “H5LTopen_file_image” section.
    INTEGER(hid_t), INTENT(IN) :: file_idWill be a file ID on success.
    INTEGER, INTENT(OUT) :: hdferrReturns the error status: 0 for success and -1 for failure.
    + + diff --git a/doxygen/examples/H5.format.html b/doxygen/examples/H5.format.html index ac1dbab683b..023c19d7c19 100644 --- a/doxygen/examples/H5.format.html +++ b/doxygen/examples/H5.format.html @@ -1432,7 +1432,7 @@

    Address of Member File N

    -

    This field Specifies the virtual address at which the member file starts.

    +

    This field specifies the virtual address at which the member file starts.

    N is the number of member files.

    @@ -17727,8 +17727,8 @@ disk address for the chunk.

    A description of the rationale that leads to the present implementation of the extensible array can be found at - - https://svn.hdfgroup.org/hdf5doc/trunk/projects/1_10_alpha/ReviseChunks/skip_lists. + + https://github.com/HDFGroup/hdf5doc/tree/master/projects/1_10_alpha/ReviseChunks/skip_lists.

    The current implementation differs from the data structure diff --git a/doxygen/examples/LibraryReleaseVersionNumbers.html b/doxygen/examples/LibraryReleaseVersionNumbers.html new file mode 100644 index 00000000000..57b211cd61b --- /dev/null +++ b/doxygen/examples/LibraryReleaseVersionNumbers.html @@ -0,0 +1,318 @@ + + +HDF5 Library Release Version Numbers + + + + +
    +

    Contents

    + + + +

    Introduction

    + +

    HDF5 software is updated on a regular basis. These updates, known +as releases, range in scope and size from small to large. Some updates +may only fix bugs, and some updates may require a change in the format +of the data file. The version numbers that are applied to updates give +information about the kinds of changes made in the updates. This Tech +Note describes what the version numbers mean.

    + +

    Note that this document describes release version numbers for the +HDF5 Library. For more information, see the +Shared Library Version Numbers section at the end of this document.

    + +

    Definitions

    + +

    Each software release of the HDF5 Library is labeled with a version + number. The version number is a set of three integers written as HDF5-1.2.3, + HDF5 version 1.2 release 3, or HDF5 Release 1.2.3. The version number + might also include text. A patch version might be labeled HDF5-1.2.3-patch1. + The '5' in "HDF5" is part of the product name and will not change during + the life of the project.

    + +

    The key components in HDF5 Library version numbers are the major version + number, the minor version number, the release number, and an optional text + string.

    + +

    The first integer in a version number is the major version + number. This integer increments when there is an extensive change + to the file format or library API. Such a change may require files to + be translated and will likely require applications to be modified.

    + +

    The second integer, 2 in the examples above, is the minor version + number. This number is incremented when there are new features that + require a change in the file format. For example, a change in file format + was required during the change from version 1.6 to version 1.8. Stable + released versions of the library are given even minor version + numbers such as 1.6 and 1.8 while odd minor version numbers such + as 1.7 and 1.9 are used on the trunk for major development. See the + section below for more information.

    + +

    The third integer, 3 in the examples above, is the release + number. A change in this number indicates that the library has + been updated. The updates might include bug fixes, performance + improvements, and new features that do not require a file format + change.

    + +

    A version number might also include some text. The two current + possibilities are patch and snap. A patch version might + be made to a released version to make available a feature or a bug + fix. In the figure below, a patch to the 1.8.5 release is labeled + 1.8.5-patch1. A snapshot is an intermediate posting of the software + in a branch or in the trunk. Snapshots are made available so that users + may begin to test changes in the software that affect their software. + The changes may range from bug fixes to new features. Snapshots are made + and released regularly. How regularly depends on whether the software + passes the tests done on each build. A possible version number for a + snapshot might be 1.9-snap81. This version would hold the 81st snapshot + off the 1.9 development branch (the current trunk). For the + snapshots are available at + https://github.com/HDFGroup/hdf5/releases/tag/snapshot.

    + +

    The Trunk, Release Branches, and Feature Branches

    + +

    The HDF Group uses a version control system to manage the HDF5 + project. Within the system, a trunk and branches are used to track + changes. The version numbers described above identify where a given + piece of software was produced in the system. The figure below shows + the general scheme.

    + + + + + + + + +
    +
    + The trunk, release branches, and feature branches
    +
    + Figure 1. The trunk, release branches, and feature branches +
    + +

    The trunk is the center of the system. New features are + implemented in feature branches and aggregated in the trunk. + Release branches are then created from the trunk.

    + +

    The minor version number of the trunk is always an odd number. From + the time of Release 1.8.0 to the first 1.10 release, the trunk will be + version 1.9. The trunk was version 1.7 from the time of release 1.6.0 + until the first 1.8 release.

    + +

    Projects that add new features, bug fixes, and performance improvements + are developed on feature branches. When a project is completed, + its feature branch is merged into the trunk. In the figure above, the + merging of a feature branch is represented by a dashed arrow from the + feature branch to the trunk. If a feature requires a file format change, + then the feature will stay in the trunk until the next significant + release. This would mean in the figure above that the new feature would + be released in a future 1.10 release branch. If a feature does not + require a file format change, then it might be merged into one or more + release branches. This would mean in the figure above that the new + feature could be merged into the 1.8 branch and could be included in + the 1.8.6 release. If the feature was added to the 1.8.5 branch, then a + patch version might be released.

    + +

    Release branches hold software that is distributed to general + users. In the figure above, a few release branches are shown below the + trunk. Work is done in release branches for a period of time. Branches + further from the trunk have less work done in them. For example, a patch + branch such as 1.8.5-patch1 may contain only one or two changes. A release + branch such as 1.8.5 may contain a number of bug fixes and new functions, + but these changes are small in number compared to the number of changes in + the 1.8 branch.

    + +

    We aim to make available to the public two maintenance releases a year. + The releases occur usually in the spring near May 15 and in the fall near + November 15. If two release branches are being maintained, then + maintenance releases may be made for each release branch. For example, + there was a time when both the 1.6 and 1.8 branches were actively + maintained. In one maintenance release, the 1.6.10 and 1.8.4 versions were + released at the same time. The 1.6 and 1.8 branches were both actively + maintained to give early adopters access to new features and to give most + users plenty of time to make the change to 1.8 software from 1.6.

    + +

    As we improve any branch, we consider the effect of any change on the + readability of objects. Applications built, for example, with version + 1.8.5 will be able to read data files written with any prior version + of the library. So, a 1.8.5 application will be able to read a dataset + written with 1.4.5. A 1.8.5 application may be able to read a dataset + written under the 1.8.7 library if no new features, features not known + to 1.8.5, were used. + +

    Version Support from the Library

    + +

    The library provides macros and functions to query and check + version numbers.

    + +

    The following constants are defined in the file H5public.h + and determine the version of the include files.

    + +
      +
    • H5_VERS_MAJOR
      • +
    • H5_VERS_MINOR
      • +
    • H5_VERS_RELEASE
      • +
    • H5_VERS_SUBRELEASE
      • +
    • H5_VERS_INFO
      • +
    + +

    The table below describes some of the function calls and macros + that can be used to query and check version numbers.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + Table 1. Version function calls and macros
    + Function Call or Macro + Comments
    H5get_libversionThis function returns through its arguments the version + numbers for the library to which the application is linked.
    H5checkThis macro uses the H5check_version function + to verify that the version number of the HDF5 include file used + to compile the application matches the version number of the + library to which the application is linked. This check occurs + automatically when the first HDF5 file is created or opened and + is important because a mismatch between the include files and the + library is likely to result in corrupted data and/or segmentation + faults. If a mismatch is detected, the library issues an error + message on the standard error stream and aborts.
    H5check_version

    This function is called by the H5check macro + with the include file version constants. The function compares + its arguments to the result returned by H5get_libversion. + If a mismatch is detected, it prints an error message on the standard + error stream and aborts.

    + +

    The behavior of this function can be modified by the + HDF5_DISABLE_VERSION_CHECK environment variable. Setting + the environment variable to a value of =911=92 will issue a + warning but continue without aborting. Setting the environment + variable to a value of =912=92 will suppress the warning + and continue silently without aborting.

    H5_VERSION_GE and + H5_VERSION_LEThese macros compare the version of the HDF5 library being used + against the version number specified in the parameters. At compile + time, they can be used to conditionally include or exclude code + based on the library's version.
    H5Pset_libver_boundsThis function can be used to control the versions of the object + formats that will be used when creating objects in a file.
    + +

    For more information on these and other function calls and macros, + see the HDF5 Reference Manual.

    + +

    Use Cases

    + +

    The purpose of this section is to describe how some of the version + functions, macros, and constants might be used.

    + +

    Application Version Checking

    + +

    Suppose first that a developer builds an application that will read + from and write to an HDF5 file. When the application is compiled, a + version of the HDF5 Library such as 1.8.6 will be used. The version + constants (H5_VERS_MAJOR, H5_VERS_MINOR, + and H5_VERS_RELEASE) are included in the application when + it is compiled.

    + +

    Suppose next that a user gets a copy of the application and starts it + up on a workstation. The executable is put into memory along with the + HDF5 Library. However, an application may only work successfully with + the version of the library with which the application was compiled. In + other words, the version of the library that is loaded when the applicati= +on + is started must be the same version as the version of the library with + which the application was compiled. This is verified by the library when + the first HDF5 API routine is called. If an application wants to confirm + early in its startup procedure that the version of the library that will + be loaded into memory at the workstation will work with the application, + then it can use the H5get_libversion and + H5check_version function calls.

    + +

    Conditional Inclusions or Exclusions Based on the Version

    + +

    The H5_VERSION_GE and H5_VERSION_LE version + macros compare the version of the HDF5 Library being used against the + version number specified in the parameters. At compile time, they can be + used to conditionally include or exclude code based on the library's + version. For example, the link functions, H5Lxxx, are + new in version 1.8, and some group functions, H5Gxxx, + are deprecated in 1.8. With the H5_VERSION_GE macro, an + application could use H5Ldelete if the library version is + 1.8.0 or greater, or it could use H5Gunlink if the library + version is less than 1.8.0.

    + +

    Specifying a Format

    +

    Suppose a data file has three datasets. It is possible that the three + datasets were added to the data file with applications using different + versions of HDF5. The different versions could be 1.4.5, 1.6.10, and + 1.8.6. If another dataset is written to the data file, then it will be + written by default in the oldest format possible that has all of the + features needed to successfully write the dataset. If a newer feature + such as compact storage, a new parameter for a function, or a partially + compressed dataset is used, then a newer format will be used. + H5Pset_libver_bounds could be used to specify the oldest + format used. In the situation above, the owners of the data file might + want all data written to the file in the future to be in a 1.8 format + rather than 1.6 or 1.4.

    + +

    Shared Library Version Numbers

    + +

    HDF5 shared libraries utilize the + +libtool versioning system in order to indicate interface +compatibility between maintenance releases of HDF5. While we always +attempt to maintain interface compatibility between minor maintenance +release versions of HDF5, if we are forced to break interface +compatibility in order to resolve a critical defect within the +library, then the library interface version attached to the shared +libraries for a given release will be incremented accordingly.

    + +

    Please note that this libtool version number for interface +compatibility is unrelated to the HDF5 release version for a given +release.

    + + + diff --git a/doxygen/examples/intro_SWMR.html b/doxygen/examples/intro_SWMR.html index f4cd586f5f8..b1adb62bdb5 100644 --- a/doxygen/examples/intro_SWMR.html +++ b/doxygen/examples/intro_SWMR.html @@ -17,8 +17,8 @@

    SWMR User's Guide

    HDF5 Library APIs