mirrors/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2026-09-25 04:09:44 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge doxygen2 into develop (#553)

Browse Source 
* Fixed warnings and started H5Epublic.h.

* Include H5FD* headers to correctly resolve references.

* Doxygen2 (#330)

* H5Eauto_is_v2.

* Added a few more calls.

* Added a few more H5E calls.

* First cut of H5E v2.

* Added the deprecated v1 calls.

* Updated spacing.

* Once more.

* Taking some inspiration from Eigen3.

* Add doxygen for the assigned functions: H5Pregister1,H5Pinsert1,H5Pen… (#352)

* Add doxygen for the assigned functions: H5Pregister1,H5Pinsert1,H5Pencode1, H5Pget_filter_by_id1,H5Pget_version, H5Pset_file_space,H5Pget_file_space. Someone already adds H5Pget_filter1. Also fixs an extra parameter 'close' call back function for HPregister2.

* doxygen work. fixs format by using clang-format.

* doxgen work for H5Pregister1 etc. Addressed Barbara and Gerd's comments.
For Quincey's comments, since we are not supposed to change the source code.
I leave this to future improvements.

* added documentation for H5P APIs (#350)

* add documenation for H5Pget_buffer,H5Pget_data_transform,H5Pget_edc_check,H5Pget_hyper_vector_size,H5Pget_preserve,H5Pget_type_conv_cb,H5Pget_vlen_mem_manager,H5Pset_btree_ratios

* format corrections

* fixed grammer

* fixed herr_t

* Better name.

* A fresh look.

* add doxygen to H5Ppublic.h

* use attention instead of warning

* Add doxygen comments in H5Ppublic.h (#375)

* Add doxygen comments in H5Ppublic.h

* H5Pset_meta_block_size
* H5Pset_metadata_read_attempts
* H5Pset_multi_type
* H5Pset_object_flush_cb
* H5Pset_sieve_buf_size
* H5Pset_small_data_block_size
* H5Pset_all_coll_metadata_ops
* H5Pget_all_coll_metadata_ops

* Add DOXYGEN_EXAMPLES_DIR to src/CMakeLists.txt

* Fix clang-format errors

* Fix filenames in doxygen/examples

* add doxygen to H5Ppublic.h (#378)

* add doxygen to H5Ppublic.h

* use attention instead of warning

Co-authored-by: Kimmy Mu <kmu@hdfgroup.org>

* Revert "add doxygen to H5Ppublic.h (#378)"

This reverts commit 2ee1821b13.

* Updated Doxygen variables.

* I forgot to copy two images.

* Enable desktop search by default.

* Add my assigned Doxygen documentation.

* Remove whitespace at EOL.  Appease clang-format.

* Addressed Chris' comments.

* Added an alias for asynchronous functions.

* One space is enough for all of us.

* Slightly restructured RM page.

* address some issues

* reformatting

* Style external links.

* reformatting

* reformatting

* Added "Metadata Caching in HDF5" as a technical note example.

* Revise this soon!

* Added specification examples.

* Fixed references.

* Added H5AC cache image stuff and file format study.

* Added older FMT versions. Where did 1.0 go?

* Updated C/C++ note and replaced ambiguous labels.

* Reformat source with clang v10.0.1.

* Added the VFL technical note.

* Added what I believe might be called version 1.0 of the format.

* Added the remaining specs.

* Added H5Z callback documentation and fixed a few mistakes.

* Added dox for deprecated H5G calls and fixed a few snippet blockIDs.

* clang-format happy?

* Ok?

* Bonus track: Deprecated H5D functions.

* Carry over the more detailed group description.

* Added documentation for the missing and deprecated H5R calls.

* Life is easier and less repetitive w/ snippets. Use them!

* Eliminate the snippet block ID artifacts in the HTML rendering.

* Fixed snippet HTML artifacts and added a few missing calls.

* Under 20 H5Ps to go!

* Almost complete!

* "This is a form of pedantry up with which I will not put." (Churchill)

* Let's not waste as much space on bulleted lists!

* First complete (?) draft of the Doxygen-based RM.

* Completeness check and minor fixes along the way.

* Pedantry.

* Adding missing H5FD calls checkpoint.

* Pedantry.

* More pedantry.

* Added H5Pset_fapl_log.

* First draft of H5ES.

* Fixed warnings.

* Prep. for map module.

* First cut of the map module.

* Pedantry.

* Possible H5F introduction.

* Fix the indentation.

* Pedantry.

* Ditto.

* Thanks to the reviewers for their comments.

* Added missing images.

* Line numbers are a distraction here.

* More examples, references, and clean-up. Don't repeat yourself!

* Clang pedantry.

* Ditto.

* More reviewer comments...

* Templatized references and cleaned up \todos.

* Committing clang-format changes

* Fixed MANIFEST.

* Addressed Quincey's comments. (OCPLs)

* Fixed a few more \todo items.

* Fixed more \todo items.

* Added attribute life cycle.

* Forgot the examples file.

* Committing clang-format changes

* Pedantry.

* Live and learn!

* Added a sample H5D life cycle.

* Committing clang-format changes

* Pedantry.

Co-authored-by: kyang2014 <kyang2014@users.noreply.github.com>
Co-authored-by: Scot Breitenfeld <brtnfld@hdfgroup.org>
Co-authored-by: Kimmy Mu <kmu@hdfgroup.org>
Co-authored-by: Christopher Hogan <ChristopherHogan@users.noreply.github.com>
Co-authored-by: jya-kmu <53388330+jya-kmu@users.noreply.github.com>
Co-authored-by: David Young <dyoung@hdfgroup.org>
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
This commit is contained in:
Gerd Heber
2021-04-26 14:07:29 -05:00
committed by GitHub
co-authored by Kimmy Mu kyang2014 Scot Breitenfeld Christopher Hogan jya-kmu David Young Larry Knox github-actions
parent 12082e728d
commit 1d680fe04c
110 changed files with 61858 additions and 1719 deletions
Download Patch File Download Diff File Expand all files Collapse all files
MANIFEST
+48 -2
View File
@@ -207,7 +207,12 @@
./doxygen/aliases
./doxygen/Doxyfile.in
./doxygen/dox/api-compat-macros.dox
./doxygen/dox/About.dox
./doxygen/dox/Cookbook.dox
./doxygen/dox/DDLBNF110.dox
./doxygen/dox/DDLBNF112.dox
./doxygen/dox/FileFormatSpec.dox
./doxygen/dox/GettingStarted.dox
./doxygen/dox/H5AC_cache_config_t.dox
./doxygen/dox/H5Acreate.dox
./doxygen/dox/H5Aiterate.dox
@@ -224,12 +229,53 @@
./doxygen/dox/H5Ovisit_by_name.dox
./doxygen/dox/H5Ovisit.dox
./doxygen/dox/H5Sencode.dox
./doxygen/dox/mainpage.dox
./doxygen/dox/MetadataCachingInHDF5.dox
./doxygen/dox/OtherSpecs.dox
./doxygen/dox/Overview.dox
./doxygen/dox/ReferenceManual.dox
./doxygen/dox/Specifications.dox
./doxygen/dox/TechnicalNotes.dox
./doxygen/dox/api-compat-macros.dox
./doxygen/dox/maybe_metadata_reads.dox
./doxygen/dox/rm-template.dox
./doxygen/examples/FF-IH_FileGroup.gif
./doxygen/examples/FF-IH_FileObject.gif
./doxygen/examples/FileFormatSpecChunkDiagram.jpg
./doxygen/examples/H5Pset_metadata_read_attempts.c
./doxygen/examples/H5Pset_object_flush_cb.c
./doxygen/examples/H5.format.1.0.html
./doxygen/examples/H5.format.1.1.html
./doxygen/examples/H5.format.2.0.html
./doxygen/examples/H5.format.html
./doxygen/examples/H5A_examples.c
./doxygen/examples/H5D_examples.c
./doxygen/examples/H5Fclose.c
./doxygen/examples/H5Fcreate.c
./doxygen/examples/H5F_examples.c
./doxygen/examples/H5Pget_metadata_read_attempts.1.c
./doxygen/examples/H5Pget_metadata_read_attempts.2.c
./doxygen/examples/H5Pget_metadata_read_attempts.3.c
./doxygen/examples/H5Pget_object_flush_cb.c
./doxygen/examples/ImageSpec.html
./doxygen/examples/PaletteExample1.gif
./doxygen/examples/Palettes.fm.anc.gif
./doxygen/examples/TableSpec.html
./doxygen/examples/ThreadSafeLibrary.html
./doxygen/examples/VFL.html
./doxygen/examples/hello_hdf5.c
./doxygen/hdf5_footer.html
./doxygen/hdf5_header.html
./doxygen/hdf5_navtree_hacks.js
./doxygen/hdf5doxy.css
./doxygen/hdf5doxy_layout.xml
./doxygen/img/FF-IH_FileGroup.gif
./doxygen/img/FF-IH_FileObject.gif
./doxygen/img/FileFormatSpecChunkDiagram.jpg
./doxygen/img/HDFG-logo.png
./doxygen/img/PaletteExample1.gif
./doxygen/img/Palettes.fm.anc.gif
./doxygen/img/ftv2node.png
./doxygen/img/ftv2pnode.png
./examples/Attributes.txt
./examples/Makefile.am
configure.ac
+21 -3
View File
@@ -95,12 +95,12 @@ AC_CONFIG_COMMANDS([pubconf], [
sed 's/#define /#define H5_/' <src/H5config.h |\
sed 's/#undef /#undef H5_/' >pubconf
if test ! -f src/H5pubconf.h; then
/bin/mv -f pubconf src/H5pubconf.h
mv -f pubconf src/H5pubconf.h
elif (diff pubconf src/H5pubconf.h >/dev/null); then
rm -f pubconf
echo "src/H5pubconf.h is unchanged"
else
/bin/mv -f pubconf src/H5pubconf.h
mv -f pubconf src/H5pubconf.h
fi
echo "Post process src/libhdf5.settings"
sed '/^#/d' < src/libhdf5.settings > libhdf5.settings.TMP
@@ -1116,16 +1116,34 @@ if test "X$HDF5_DOXYGEN" = "Xyes"; then
AC_SUBST([DOXYGEN_OPTIMIZE_OUTPUT_FOR_C])
AC_SUBST([DOXYGEN_MACRO_EXPANSION])
AC_SUBST([DOXYGEN_OUTPUT_DIRECTORY])
AC_SUBST([DOXYGEN_EXAMPLES_DIRECTORY])
AC_SUBST([DOXYGEN_LAYOUT_FILE])
AC_SUBST([DOXYGEN_HTML_HEADER])
AC_SUBST([DOXYGEN_HTML_FOOTER])
AC_SUBST([DOXYGEN_HTML_EXTRA_STYLESHEET])
AC_SUBST([DOXYGEN_HTML_EXTRA_FILES])
AC_SUBST([DOXYGEN_SERVER_BASED_SEARCH])
AC_SUBST([DOXYGEN_EXTERNAL_SEARCH])
AC_SUBST([DOXYGEN_SEARCHENGINE_URL])
DOXYGEN_PACKAGE=${PACKAGE_NAME}
DOXYGEN_VERSION_STRING=${PACKAGE_VERSION}
DOXYGEN_INCLUDE_ALIASES='$(SRCDIR)/doxygen/aliases'
DOXYGEN_PROJECT_LOGO='$(SRCDIR)/doxygen/img/HDFG-logo.png'
DOXYGEN_PROJECT_BRIEF="C-API Reference"
DOXYGEN_PROJECT_BRIEF=
DOXYGEN_INPUT_DIRECTORY='$(SRCDIR) $(SRCDIR)/doxygen/dox'
DOXYGEN_OPTIMIZE_OUTPUT_FOR_C=YES
DOXYGEN_MACRO_EXPANSION=YES
DOXYGEN_OUTPUT_DIRECTORY=hdf5lib_docs
DOXYGEN_EXAMPLES_DIRECTORY='$(SRCDIR)/doxygen/examples'
DOXYGEN_LAYOUT_FILE='$(SRCDIR)/doxygen/hdf5doxy_layout.xml'
DOXYGEN_HTML_HEADER='$(SRCDIR)/doxygen/hdf5_header.html'
DOXYGEN_HTML_FOOTER='$(SRCDIR)/doxygen/hdf5_footer.html'
DOXYGEN_HTML_EXTRA_STYLESHEET='$(SRCDIR)/doxygen/hdf5doxy.css'
DOXYGEN_HTML_EXTRA_FILES='$(SRCDIR)/doxygen/hdf5_navtree_hacks.js $(SRCDIR)/doxygen/img/ftv2node.png $(SRCDIR)/doxygen/img/ftv2pnode.png'
DOXYGEN_SERVER_BASED_SEARCH=NO
DOXYGEN_EXTERNAL_SEARCH=NO
DOXYGEN_SEARCHENGINE_URL=
DX_INIT_DOXYGEN([HDF5], [../doxygen/Doxyfile], [hdf5lib_docs])
doxygen/Doxyfile.in
+22 -13
View File
@@ -738,7 +738,7 @@ FILE_VERSION_FILTER =
# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE
# tag is left empty.
LAYOUT_FILE =
LAYOUT_FILE = @DOXYGEN_LAYOUT_FILE@
# The CITE_BIB_FILES tag can be used to specify one or more bib files containing
# the reference definitions. This must be a list of .bib files. The .bib
@@ -855,7 +855,16 @@ INPUT_ENCODING = UTF-8
FILE_PATTERNS = H5*public.h \
H5*module.h \
H5FDcore.h \
H5FDdirect.h \
H5FDfamily.h \
H5FDlog.h \
H5FDmpi.h \
H5FDmpio.h \
H5FDmulti.h \
H5FDsec2.h \
H5FDstdio.h \
H5FDwindows.h \
H5VLconnector.h \
H5VLconnector_passthru.h \
H5VLnative.h \
@@ -908,7 +917,7 @@ EXCLUDE_SYMBOLS =
# that contain example code fragments that are included (see the \include
# command).
EXAMPLE_PATH = ../src ../examples ../test examples
EXAMPLE_PATH = ../src ../examples ../test @DOXYGEN_EXAMPLES_DIRECTORY@
# If the value of the EXAMPLE_PATH tag contains directories, you can use the
# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and
@@ -1169,7 +1178,7 @@ HTML_FILE_EXTENSION = .html
# of the possible markers and block names see the documentation.
# This tag requires that the tag GENERATE_HTML is set to YES.
HTML_HEADER =
HTML_HEADER = @DOXYGEN_HTML_HEADER@
# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each
# generated HTML page. If the tag is left blank doxygen will generate a standard
@@ -1179,7 +1188,7 @@ HTML_HEADER =
# that doxygen normally uses.
# This tag requires that the tag GENERATE_HTML is set to YES.
HTML_FOOTER =
HTML_FOOTER = @DOXYGEN_HTML_FOOTER@
# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style
# sheet that is used by each HTML page. It can be used to fine-tune the look of
@@ -1204,7 +1213,7 @@ HTML_STYLESHEET =
# list). For an example see the documentation.
# This tag requires that the tag GENERATE_HTML is set to YES.
HTML_EXTRA_STYLESHEET =
HTML_EXTRA_STYLESHEET = @DOXYGEN_HTML_EXTRA_STYLESHEET@
# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or
# other source files which should be copied to the HTML output directory. Note
@@ -1214,7 +1223,7 @@ HTML_EXTRA_STYLESHEET =
# files will be copied as-is; there are no commands or markers available.
# This tag requires that the tag GENERATE_HTML is set to YES.
HTML_EXTRA_FILES =
HTML_EXTRA_FILES = @DOXYGEN_HTML_EXTRA_FILES@
# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen
# will adjust the colors in the style sheet and background images according to
@@ -1272,7 +1281,7 @@ HTML_DYNAMIC_MENUS = NO
# The default value is: NO.
# This tag requires that the tag GENERATE_HTML is set to YES.
HTML_DYNAMIC_SECTIONS = NO
HTML_DYNAMIC_SECTIONS = YES
# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries
# shown in the various tree structured indices initially; the user can expand
@@ -1484,7 +1493,7 @@ ECLIPSE_DOC_ID = org.doxygen.Project
# The default value is: NO.
# This tag requires that the tag GENERATE_HTML is set to YES.
DISABLE_INDEX = NO
DISABLE_INDEX = YES
# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index
# structure should be generated to display hierarchical information. If the tag
@@ -1632,7 +1641,7 @@ MATHJAX_CODEFILE =
# The default value is: YES.
# This tag requires that the tag GENERATE_HTML is set to YES.
SEARCHENGINE = NO
SEARCHENGINE = YES
# When the SERVER_BASED_SEARCH tag is enabled the search engine will be
# implemented using a web server instead of a web client using JavaScript. There
@@ -1644,7 +1653,7 @@ SEARCHENGINE = NO
# The default value is: NO.
# This tag requires that the tag SEARCHENGINE is set to YES.
SERVER_BASED_SEARCH = YES
SERVER_BASED_SEARCH = @DOXYGEN_SERVER_BASED_SEARCH@
# When EXTERNAL_SEARCH tag is enabled doxygen will no longer generate the PHP
# script for searching. Instead the search results are written to an XML file
@@ -1660,7 +1669,7 @@ SERVER_BASED_SEARCH = YES
# The default value is: NO.
# This tag requires that the tag SEARCHENGINE is set to YES.
EXTERNAL_SEARCH = NO
EXTERNAL_SEARCH = @DOXYGEN_EXTERNAL_SEARCH@
# The SEARCHENGINE_URL should point to a search engine hosted by a web server
# which will return the search results when EXTERNAL_SEARCH is enabled.
@@ -1671,7 +1680,7 @@ EXTERNAL_SEARCH = NO
# Searching" for details.
# This tag requires that the tag SEARCHENGINE is set to YES.
SEARCHENGINE_URL =
SEARCHENGINE_URL = @DOXYGEN_SEARCHENGINE_URL@
# When SERVER_BASED_SEARCH and EXTERNAL_SEARCH are both enabled the unindexed
# search data is written to a file for indexing by an external tool. With the
@@ -2168,7 +2177,7 @@ INCLUDE_FILE_PATTERNS =
# recursively expanded use the := operator instead of the = operator.
# This tag requires that the tag ENABLE_PREPROCESSING is set to YES.
PREDEFINED =
PREDEFINED = H5_HAVE_PARALLEL
# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then this
# tag can be used to specify a list of macro names that should be expanded. The
doxygen/aliases
+61 -2
View File
@@ -1,3 +1,5 @@
ALIASES += THG="The HDF Group"
################################################################################
# Styling
################################################################################
@@ -35,6 +37,15 @@ ALIASES += op{1}="\param[in] \1 Callback function"
ALIASES += op_data="\param[in,out] op_data User-defined callback function context"
ALIASES += op_data{1}="\param[in,out] \1 User-defined callback function context"
ALIASES += op_data_in="\param[in] op_data User-defined callback function context"
ALIASES += op_data_in{1}="\param[in] \1 User-defined callback function context"
################################################################################
# Asynchronous
################################################################################
ALIASES += async_variant_of{1}="Asynchronous version of \1()"
################################################################################
# Attributes
################################################################################
@@ -66,6 +77,13 @@ ALIASES += type_id{1}="\param[in] \1 Datatype identifier"
ALIASES += file_type_id{1}="\param[in] \1 Datatype (in-file) identifier"
ALIASES += mem_type_id{1}="\param[in] \1 Datatype (in-memory) identifier"
################################################################################
# Errors
################################################################################
ALIASES += estack_id="\param[in] estack_id Error stack identifier"
ALIASES += estack_id{1}="\param[in] \1 Error stack identifier"
################################################################################
# Files
################################################################################
@@ -102,6 +120,13 @@ ALIASES += fgdta_loc_id{1}="\loc_id{\1}. The identifier may be that of a file, g
ALIASES += fg_loc_id="\loc_id. The identifier may be that of a file or group."
ALIASES += fg_loc_id{1}="\loc_id{\1}. The identifier may be that of a file or group."
################################################################################
# Maps
################################################################################
ALIASES += map_id="\param[in] map_id Map identifier"
ALIASES += map_id{1}="\param[in] \1 Map identifier"
################################################################################
# Property lists
################################################################################
@@ -121,6 +146,9 @@ ALIASES += dcpl_id{1}="\param[in] \1 Dataset creation property list identifier"
ALIASES += dxpl_id="\param[in] dxpl_id Dataset transfer property list identifier"
ALIASES += dxpl_id{1}="\param[in] \1 Dataset transfer property list identifier"
ALIASES += gacpl_id="\param[in] plist_id File, group, dataset, datatype, link, or attribute access property list identifier"
ALIASES += gacpl_id{1}="\param[in] \1 File, group, dataset, datatype, link, or attribute access property list identifier"
ALIASES += gapl_id="\param[in] gapl_id Group access property list identifier"
ALIASES += gapl_id{1}="\param[in] \1 Group access property list identifier"
@@ -133,9 +161,18 @@ ALIASES += lapl_id{1}="\param[in] \1 Link access property list identifier"
ALIASES += lcpl_id="\param[in] lcpl_id Link creation property list identifier"
ALIASES += lcpl_id{1}="\param[in] \1 Link creation property list identifier"
ALIASES += mapl_id="\param[in] mapl_id Map access property list identifier"
ALIASES += mapl_id{1}="\param[in] \1 Map access property list identifier"
ALIASES += mcpl_id="\param[in] mcpl_id Map creation property list identifier"
ALIASES += mcpl_id{1}="\param[in] \1 Map creation property list identifier"
ALIASES += oapl_id="\param[in] oapl_id Object access property list identifier"
ALIASES += oapl_id{1}="\param[in] \1 Object access property list identifier"
ALIASES += ocpl_id="\param[in] oapl_id Object creation property list identifier"
ALIASES += ocpl_id{1}="\param[in] \1 Object creation property list identifier"
ALIASES += plist_id="\param[in] plist_id Property list identifier"
ALIASES += plist_id{1}="\param[in] \1 Property list identifier"
@@ -173,7 +210,8 @@ ALIASES += fgdta_loc_obj_id{1}="\loc_obj_id{\1}. The identifier may be that of a
ALIASES += app_file="\param[in] app_file For internal use only, not a visible user parameter"
ALIASES += app_func="\param[in] app_func For internal use only, not a visible user parameter"
ALIASES += app_line="\param[in] app_line For internal use only, not a visible user parameter"
ALIASES += es_id="\param[in] es_id The event set ID to add this asynchronous operation to. H5ES_NONE may be used for synchronous execution."
ALIASES += es_id="\param[in] es_id Event set identifier"
ALIASES += es_id{1}="\param[in] \1 Event set identifier"
################################################################################
# Others
@@ -181,11 +219,32 @@ ALIASES += es_id="\param[in] es_id The event set ID to add this asynchronous ope
ALIASES += estack_id="\param[in] estack_id Error stack identifier"
ALIASES += estack_id{1}="\param[in] \1 Error stack identifier"
ALIASES += cpp_c_api_note="\attention \Bold{C++ Developers using HDF5 C-API functions beware:}\n Several functions in this C-API take function pointers or callbacks as arguments. Examples include H5Pset_elink_cb(), H5Pset_type_conv_cb(), H5Tconvert(), and H5Ewalk2(). Application code must ensure that those callback functions return normally such to allow the HDF5 to manage its resources and maintain a consistent state. For instance, those functions must not use the C \c setjmp / \c longjmp mechanism to leave those callback functions. Within the context of C++, any exceptions thrown within the callback function must be caught, such as with a \Code{catch(…)} statement. Any exception state can be placed within the provided user data function call arguments, and may be thrown again once the calling function has returned. Exceptions raised and not handled inside the callback are not supported as it might leave the HDF5 library in an inconsistent state. Similarly, using C++20 coroutines cannot be used as callbacks, since they do not support plain return statements. If a callback function yields execution to another C++20 coroutine calling HDF5 functions as well, this may lead to undefined behavior."
ALIASES += sa_metadata_ops="\sa \li H5Pget_all_coll_metadata_ops() \li H5Pget_coll_metadata_write() \li H5Pset_all_coll_metadata_ops() \li H5Pset_coll_metadata_write() \li \ref maybe_metadata_reads"
################################################################################
# References
################################################################################
ALIASES += ref_cons_semantics="<a href=\"https://portal.hdfgroup.org/display/HDF5/Enabling+a+Strict+Consistency+Semantics+Model+in+Parallel+HDF5\">Enabling a Strict Consistency Semantics Model in Parallel HDF5</a>"
ALIASES += ref_dld_filters="<a href=\"https://portal.hdfgroup.org/display/HDF5/HDF5+Dynamically+Loaded+Filters\">HDF5 Dynamically Loaded Filters</a>"
ALIASES += ref_file_image_ops="<a href=\"https://portal.hdfgroup.org/display/HDF5/HDF5+File+Image+Operations\">HDF5 File Image Operations</a>"
ALIASES += ref_filter_pipe="<a href=\"https://portal.hdfgroup.org/display/HDF5/HDF5+Data+Flow+Pipeline+for+H5Dread\">Data Flow Pipeline for H5Dread()</a>"
ALIASES += ref_group_impls="<a href=\"https://portal.hdfgroup.org/display/HDF5/Groups\">Group implementations in HDF5</a>"
ALIASES += ref_h5lib_relver="<a href=\"https://portal.hdfgroup.org/display/HDF5/HDF5+Library+Release+Version+Numbers\">HDF5 Library Release Version Numbers</a>"
ALIASES += ref_mdc_in_hdf5="<a href=\"https://portal.hdfgroup.org/display/HDF5/Metadata+Caching+in+HDF5\">Metadata Caching in HDF5</a>"
ALIASES += ref_mdc_logging="<a href=\"https://portal.hdfgroup.org/display/HDF5/H5F_START_MDC_LOGGING\">Metadata Cache Logging</a>"
ALIASES += ref_news_112="<a href=\"https://portal.hdfgroup.org/display/HDF5/New+Features+in+HDF5+Release+1.12\">New Features in HDF5 Release 1.12</a>"
ALIASES += ref_h5ocopy="<a href=\"https://portal.hdfgroup.org/display/HDF5/Copying+Committed+Datatypes+with+H5Ocopy\">Copying Committed Datatypes with H5Ocopy()</a>"
ALIASES += ref_sencode_fmt_change="<a href=\"https://portal.hdfgroup.org/pages/viewpage.action?pageId=58100093&preview=/58100093/58100094/encode_format_RFC.pdf\">RFC H5Secnode() / H5Sdecode() Format Change</a>"
ALIASES += ref_vlen_strings="\Emph{Creating variable-length string datatypes}"
ALIASES += ref_vol_doc="VOL documentation"
################################################################################
# The Usual Suspects
################################################################################
ALIASES += click4more="(Click on a enumerator, field, or type for more information.)"
ALIASES += csets="<table><tr><td>#H5T_CSET_ASCII</td><td>US ASCII</td></tr><tr><td>#H5T_CSET_UTF8</td><td>UTF-8 Unicode encoding</td></tr></table>"
ALIASES += datatype_class=" \li #H5T_INTEGER \li #H5T_FLOAT \li #H5T_STRING \li #H5T_BITFIELD \li #H5T_OPAQUE \li #H5T_COMPOUND \li #H5T_REFERENCE \li #H5T_ENUM \li #H5T_VLEN \li #H5T_ARRAY"
ALIASES += file_access="<table><tr><td>#H5F_ACC_RDWR</td><td>File was opened with read/write access.</td></tr><tr><td>#H5F_ACC_RDONLY</td><td>File was opened with read-only access.</td></tr><tr><td>#H5F_ACC_SWMR_WRITE</td><td>File was opened with read/write access for a single-writer/multiple-reader (SWMR) scenario. Note that the writer process must also open the file with the #H5F_ACC_RDWR flag.</td></tr><tr><td>#H5F_ACC_SWMR_READ</td><td>File was opened with read-only access for a single-writer/multiple-reader (SWMR) scenario. Note that the reader process must also open the file with the #H5F_ACC_RDONLY flag.</td></tr></table>"
@@ -201,5 +260,5 @@ ALIASES += scopes="<table><tr><td>#H5F_SCOPE_GLOBAL</td><td>Flushes the entire v
ALIASES += sign_prop="<table><tr><td>#H5T_SGN_NONE</td><td>0</td><td>Unsigned integer type</td></tr><tr><td>#H5T_SGN_2</td><td>1</td><td>Two's complement signed integer type</td></tr></table>"
ALIASES += storage_type="<table><tr><td>#H5G_STORAGE_TYPE_COMPACT</td><td>Compact storage</td></tr><tr><td>#H5G_STORAGE_TYPE_DENSE</td><td>Indexed storage</td></tr><tr><td>#H5G_STORAGE_TYPE_SYMBOL_TABLE</td><td>Symbol tables, the original HDF5 structure</td></tr></table>"
ALIASES += str_pad_type="<table><tr><td>#H5T_STR_NULLTERM</td><td>0</td><td>Null terminate (as C does)</td></tr><tr><td>#H5T_STR_NULLPAD</td><td>1</td><td>Pad with zeros</td></tr><tr><td>#H5T_STR_SPACEPAD</td><td>2</td><td>Pad with spaces (as FORTRAN does)</td></tr></table>"
ALIASES += virtual=" \see Supporting Functions: \li H5Pget_layout() \li H5Pset_layout() \li H5Sget_regular_hyperslab() \li H5Sis_regular_hyperslab() \li H5Sselect_hyperslab() \see VDS Functions: \li H5Pget_virtual_count() \li H5Pget_virtual_dsetname() \li H5Pget_virtual_filename() \li H5Pget_virtual_prefix() \li H5Pget_virtual_printf_gap() \li H5Pget_virtual_srcspace() \li H5Pget_virtual_view() \li H5Pget_virtual_vspace() \li H5Pset_virtual \li H5Pset_virtual_prefix() \li H5Pset_virtual_printf_gap() \li H5Pset_virtual_view()"
ALIASES += see_virtual=" \see Supporting Functions: H5Pget_layout(), H5Pset_layout(), H5Sget_regular_hyperslab(), H5Sis_regular_hyperslab(), H5Sselect_hyperslab() \see VDS Functions: H5Pget_virtual_count(), H5Pget_virtual_dsetname(), H5Pget_virtual_filename(), H5Pget_virtual_prefix(), H5Pget_virtual_printf_gap(), H5Pget_virtual_srcspace(), H5Pget_virtual_view(), H5Pget_virtual_vspace(), H5Pset_virtual(), H5Pset_virtual_prefix(), H5Pset_virtual_printf_gap(), H5Pset_virtual_view()"
ALIASES += obj_info_fields="<table><tr><th>Flag</th><th>Purpose</th></tr><tr><td>#H5O_INFO_BASIC</td><td>Fill in the fileno, addr, type, and rc fields</td></tr><tr> <td>#H5O_INFO_TIME</td><td>Fill in the atime, mtime, ctime, and btime fields</td></tr><tr> <td>#H5O_INFO_NUM_ATTRS</td> <td>Fill in the num_attrs field</td></tr><tr><td>#H5O_INFO_HDR</td><td>Fill in the num_attrs field</td></tr><tr><td>#H5O_INFO_META_SIZE</td><td>Fill in the meta_size field</td></tr><tr><td>#H5O_INFO_ALL</td><td>#H5O_INFO_BASIC | #H5O_INFO_TIME | #H5O_INFO_NUM_ATTRS | #H5O_INFO_HDR | #H5O_INFO_META_SIZE</td></tr></table>"
doxygen/dox/About.dox
+11
View File
@@ -0,0 +1,11 @@
/** \page About About
The implementation of this documentation set is based on the fantastic work of the
<a href="https://eigen.tuxfamily.org/index.php?title=Main_Page">Eigen project</a>.
Please refer to their <a href="https://gitlab.com/libeigen/eigen">GitLab repository</a>
and the online version of their
<a href="http://eigen.tuxfamily.org/dox/">Doxygen-based documentation</a>.
Not only does Eigen set a standard as a piece of software, but also as an example
of <em>documentation done right</em>.
*/
doxygen/dox/Cookbook.dox
+5
View File
@@ -0,0 +1,5 @@
/** \page Cookbook Cookbook
Healthy, everyday recipes for every taste and budget...
*/
doxygen/dox/DDLBNF110.dox
+650
View File
@@ -0,0 +1,650 @@
/** \page DDLBNF110 DDL in BNF through HDF5 1.10
\todo Revise this & break it up!
\section intro110 Introduction
This document contains the data description language (DDL) for an HDF5 file. The
description is in Backus-Naur Form (BNF).
\section expo110 Explanation of Symbols
This section contains a brief explanation of the symbols used in the DDL.
\code{.unparsed}
::= defined as
<tname> a token with the name tname
<a> | <b> one of <a> or <b>
<a>opt zero or one occurrence of <a>
<a>* zero or more occurrence of <a>
<a>+ one or more occurrence of <a>
[0-9] an element in the range between 0 and 9
'[' the token within the quotes (used for special characters)
TBD To Be Decided
\endcode
\section ddl110 The DDL
\code{.unparsed}
<file> ::= HDF5 <file_name> { <file_super_block>opt <root_group> }
<file_name> ::= <identifier>
<file_super_block> ::= SUPER_BLOCK {
SUPERBLOCK_VERSION <int_value>
FREELIST_VERSION <int_value>
SYMBOLTABLE_VERSION <int_value>
OBJECTHEADER_VERSION <int_value>
OFFSET_SIZE <int_value>
LENGTH_SIZE <int_value>
BTREE_RANK <int_value>
BTREE_LEAF <int_value>
ISTORE_K <int_value>
<super_block_filespace>
USER_BLOCK {
USERBLOCK_SIZE <int_value>
}
}
<super_block_filespace> ::= FILE_SPACE_STRATEGY <super_block_strategy>
FREE_SPACE_PERSIST <boolean_value>
FREE_SPACE_SECTION_THRESHOLD <int_value>
FILE_SPACE_PAGE_SIZE <int_value>
<super_block_strategy> ::= H5F_FSPACE_STRATEGY_FSM_AGGR | H5F_FSPACE_STRATEGY_PAGE |
H5F_FSPACE_STRATEGY_AGGR | H5F_FSPACE_STRATEGY_NONE |
Unknown strategy
<root_group> ::= GROUP "/" {
<anon_named_datatype>*
<object_id>opt
<group_comment>opt
<group_attribute>*
<group_member>*
}
<datatype> ::= <atomic_type> | <compound_type> | <variable_length_type> | <array_type>
<anon_named_datatype> ::= DATATYPE <anon_named_type_name> {
<datatype>
}
<anon_named_type_name> ::= the assigned name for anonymous named type is
in the form of #oid, where oid is the object id
of the type
<atomic_type> ::= <integer> | <float> | <time> | <string> |
<bitfield> | <opaque> | <reference> | <enum>
<boolean_value> ::= FALSE | TRUE
<integer> ::= H5T_STD_I8BE | H5T_STD_I8LE |
H5T_STD_I16BE | H5T_STD_I16LE |
H5T_STD_I32BE | H5T_STD_I32LE |
H5T_STD_I64BE | H5T_STD_I64LE |
H5T_STD_U8BE | H5T_STD_U8LE |
H5T_STD_U16BE | H5T_STD_U16LE |
H5T_STD_U32BE | H5T_STD_U32LE |
H5T_STD_U64BE | H5T_STD_U64LE |
H5T_NATIVE_CHAR | H5T_NATIVE_UCHAR |
H5T_NATIVE_SHORT | H5T_NATIVE_USHORT |
H5T_NATIVE_INT | H5T_NATIVE_UINT |
H5T_NATIVE_LONG | H5T_NATIVE_ULONG |
H5T_NATIVE_LLONG | H5T_NATIVE_ULLONG
<float> ::= H5T_IEEE_F32BE | H5T_IEEE_F32LE |
H5T_IEEE_F64BE | H5T_IEEE_F64LE |
H5T_NATIVE_FLOAT | H5T_NATIVE_DOUBLE |
H5T_NATIVE_LDOUBLE
<time> ::= H5T_TIME: not yet implemented
<string> ::= H5T_STRING {
STRSIZE <strsize>;
STRPAD <strpad>;
CSET <cset>;
CTYPE <ctype>;
}
<strsize> ::= <int_value>
<strpad> ::= H5T_STR_NULLTERM | H5T_STR_NULLPAD | H5T_STR_SPACEPAD
<cset> ::= H5T_CSET_ASCII | H5T_CSET_UTF8
<ctype> ::= H5T_C_S1 | H5T_FORTRAN_S1
<bitfield> ::= H5T_STD_B8BE | H5T_STD_B8LE |
H5T_STD_B16BE | H5T_STD_B16LE |
H5T_STD_B32BE | H5T_STD_B32LE |
H5T_STD_B64BE | H5T_STD_B64LE
<opaque> ::= H5T_OPAQUE {
OPAQUE_TAG <identifier>;
OPAQUE_SIZE <int_value>;opt
}
<reference> ::= H5T_REFERENCE { <ref_type> }
<ref_type> ::= H5T_STD_REF_OBJECT | H5T_STD_REF_DSETREG | H5T_STD_REF | UNDEFINED
<compound_type> ::= H5T_COMPOUND {
<member_type_def>+
}
<member_type_def> ::= <datatype> <field_name>;
<field_name> ::= <identifier>
<variable_length_type> ::= H5T_VLEN { <datatype> }
<array_type> ::= H5T_ARRAY { <dim_sizes> <datatype> }
<dim_sizes> ::= '['<dimsize>']' | '['<dimsize>']'<dim_sizes>
<dimsize> ::= <int_value>
<attribute> ::= ATTRIBUTE <attr_name> {
<dataset_type>
<dataset_space>
<data>opt
}
<attr_name> ::= <identifier>
<dataset_type> ::= DATATYPE <path_name> | <datatype>
<enum> ::= H5T_ENUM {
<enum_base_type> <enum_def>+
}
<enum_base_type> ::= <integer>
// Currently enums can only hold integer type data, but they may be expanded
// in the future to hold any datatype
<enum_def> ::= <enum_symbol> <enum_val>;
<enum_symbol> ::= <identifier>
<enum_val> ::= <int_value>
<path_name> ::= <path_part>+
<path_part> ::= /<identifier>
<dataspace> ::= <scalar_space> | <simple_space> | <complex_space> | <null_space>
<null_space> ::= NULL
<scalar_space> ::= SCALAR
<simple_space> ::= SIMPLE { <current_dims> / <max_dims> }
<complex_space> ::= COMPLEX { <complex_space_definition> }
<dataset_space> ::= DATASPACE <path_name> | <dataspace>
<current_dims> ::= <dims>
<max_dims> ::= '(' <max_dim_list> ')'
<max_dim_list> ::= <max_dim> | <max_dim>, <max_dim_list>
<max_dim> ::= <int_value> | H5S_UNLIMITED
<data> ::= <subset> | <data_values>
<data_values> ::= DATA {
<scalar_space_data> | <simple_space_data>
}
<scalar_space_data> ::= <any_element>
<any_element> ::= <atomic_element> | <compound_element> |
<variable_length_element> | <array_element>
<any_data_seq> ::= <any_element> | <any_element>, <any_data_seq>
<atomic_element> :: = <integer_data> | <float_data> | <time_data> |
<string_data> | <bitfield_data> | <opaque_data> |
<enum_data> | <reference_data>
<subset> ::= SUBSET {
<start>;
<stride>;
<count>;
<block>;
DATA {
<simple_space_data>
}
}
<start> ::= START (<coor_list>)
<stride> ::= STRIDE (<pos_list>)
<count> ::= COUNT (<max_dim_list>)
<block> ::= BLOCK (<max_dim_list>)
<coor_list> ::= <coor_data>, <coor_list> | <coor_data>
<coor_data> ::= <integer_data> | H5S_UNLIMITED
<integer_data> ::= <int_value>
<float_data> ::= a floating point number
<time_data> ::= DATA{ not yet implemented.}
<string_data> ::= a string
// A string is enclosed in double quotes.
// If a string is displayed on more than one line, string concatenate
// operator '//'is used.
<bitfield_data> ::= <hex_value>
<opaque_data> ::= <hex_value>:<hex_value> | <hex_value>
<enum_data> ::= <enum_symbol>
<reference_data> ::= <object_ref_data> | <data_region_data> | <attribute_data> | NULL
<object_ref_data> ::= <object_type> <object_num>
<object_type> ::= DATASET | GROUP | DATATYPE
<object_id> ::= OBJECTID { <object_num> }
<object_num> ::= <int_value>:<int_value> | <int_value>
<attribute_data> ::= ATTRIBUTE <attr_name>
<data_region_data> ::= DATASET <dataset_name> {
<data_region_type>opt <data_region_data_list>
<dataset_type>opt <dataset_space>opt
<data>opt
}
<data_region_type> ::= REGION_TYPE <data_region_data_type>
<data_region_data_type> ::= POINT | BLOCK
<data_region_data_list> ::= <data_region_data_info>, <data_region_data_list> |
<data_region_data_info>
<data_region_data_info> ::= <region_info> | <point_info>
<region_info> ::= (<lower_region_vals>)-(<upper_region_vals>)
<lower_region_vals> ::= <lower_bound>, <lower_region_vals> | <lower_bound>
<upper_region_vals> ::= <upper_bound>, <upper_region_vals> | <upper_bound>
<lower_bound> ::= <int_value>
<upper_bound> ::= <int_value>
<point_info> ::= (<point_vals>)
<point_vals> ::= <int_value> | <int_value>, <point_vals>
<compound_element> ::= { <any_data_seq> }
<atomic_simple_data> :: = <atomic_element>, <atomic_simple_data> |
<atomic_element>
<simple_space_data> :: = <any_data_seq>
<variable_length_element> ::= ( <any_data_seq> )
<array_element> ::= '[' <any_data_seq> ']'
<named_datatype> ::= DATATYPE <type_name> { <datatype> }
<type_name> ::= <identifier>
<hardlink> ::= HARDLINK <path_name>
<group> ::= GROUP <group_name> { <hardlink> | <group_info> }
<group_comment> ::= COMMENT <string_data>
<group_name> ::= <identifier>
<group_info> ::= <object_id>opt <group_comment>opt <group_attribute>*
<group_member>*
<group_attribute> ::= <attribute>
<group_member> ::= <named_datatype> | <group> | <dataset> |
<softlink> | <external_link>
<dataset> ::= DATASET <dataset_name> { <hardlink> | <dataset_info> }
<dataset_info> ::= <dataset_type>
<dataset_space>
<dcpl_info>opt
<dataset_attribute>* <object_id>opt
<data>opt
// Tokens above can be in any order as long as <data> is
// after <dataset_type> and <dataset_space>.
<dcpl_info> ::= <storagelayout>
<compression_filters>
<fillvalue>
<allocationtime>
<dataset_name> ::= <identifier>
<storagelayout> :: = STORAGE_LAYOUT {
<contiguous_layout> | <chunked_layout> |
<compact_layout> | <virtual_layout>
}
<contiguous_layout> ::= CONTIGUOUS
<internal_layout> | <external_layout>
<chunked_layout> ::= CHUNKED <dims>
<filter_ratio>opt
<compact_layout> ::= COMPACT
<size>
<internal_layout> ::= <size>
<offset>
<external_layout> ::= EXTERNAL {
<external_file>+
}
<virtual_layout> ::= <vmaps>*opt
<vmaps> ::= MAPPING <int_value> {
<virtual_map>
<source_map>
}
<virtual_map> ::= VIRTUAL {
<vmaps_selection>
}
<source_map> ::= SOURCE {
FILE <file_name>
DATASET <dataset_name>
<vmaps_selection>
}
<vmaps_selection> ::= <regular_hyperslab> | <irregular_hyperslab> |
<select_points> | <select_none> | <select_all>
<regular_hyperslab> ::= SELECTION REGULAR_HYPERSLAB {
<start>
<stride>
<count>
<block>
}
<irregular_hyperslab> ::= SELECTION IRREGULAR_HYPERSLAB {
<region_info>+
}
<select_points> ::= SELECTION POINT {
(<coor_list>)+
}
<select_none> ::= SELECTION NONE
<select_all> ::= SELECTION ALL
<dims> ::= (<dims_values>)
<dims_values> ::= <int_value> | <int_value>, <dims_values>
<external_file> ::= FILENAME <file_name> <size> <offset>
<offset> ::= OFFSET <int_value>
<size> ::= SIZE <int_value>
<filter_ratio> ::= <size> | <compressionratio>
<compressionratio> :: = <size> (<float_data>:1 COMPRESSION)
<compression_filters> :: = FILTERS {
<filter_type>+ | NONE
}
<filter_type> :: = <filter_deflate> | <filter_shuffle> |
<filter_flecther> | <filter_szip> |
<filter_nbit> | <filter_scaleoffset> |
<filter_default>
<filter_default> :: = <filter_user> {
FILTER_ID <int_value>
<filter_comment>opt
<filter_params>opt
}
<filter_user> :: = USER_DEFINED_FILTER
<filter_deflate> :: = COMPRESSION DEFLATE { LEVEL <int_value> }
<filter_shuffle> :: = PREPROCESSING SHUFFLE
<filter_flecther> :: = CHECKSUM FLETCHER32
<filter_szip> :: = COMPRESSION SZIP {
PIXELS_PER_BLOCK <int_value>
<filter_szip_mode>opt
<filter_szip_coding>opt
<filter_szip_order>opt
<filter_szip_header>opt
}
<filter_szip_mode> :: = MODE HARDWARE | K13
<filter_szip_coding> :: = CODING ENTROPY | NEAREST NEIGHBOUR
<filter_szip_order> :: = BYTE_ORDER LSB | MSB
<filter_szip_header> :: = HEADER RAW
<filter_nbit> :: = CHECKSUM NBIT
<filter_scaleoffset> :: = COMPRESSION SCALEOFFSET { MIN BITS <int_value> }
<filter_comment> :: = COMMENT <identifier>
<filter_params> :: = PARAMS { <int_value>* }
<fillvalue> ::= FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC | H5D_FILL_TIME_NEVER | H5D_FILL_TIME_IFSET
VALUE H5D_FILL_VALUE_UNDEFINED | H5D_FILL_VALUE_DEFAULT | <any_element>
}
<allocationtime> ::= ALLOCATION_TIME {
H5D_ALLOC_TIME_EARLY | H5D_ALLOC_TIME_INCR |
H5D_ALLOC_TIME_LATE
}
<dataset_attribute> ::= <attribute>
<softlink> ::= SOFTLINK <softlink_name> {
LINKTARGET <target>
}
<softlink_name> ::= <identifier>
<target> ::= <identifier>
<external_link> ::= EXTERNAL_LINK <external_link_name> {
TARGETFILE <targetfile>
TARGETPATH <targetpath> <targetobj>opt
}
<external_link_name> ::= <identifier>
<user_defined_link> ::= USERDEFINED_LINK <external_link_name> {
LINKCLASS <user_link_type>
}
<user_link_type> ::= <int_value>
<targetfile> ::= <file_name>
<targetpath> ::= <identifier>
<targetobj> ::= <named_datatype> | <group> | <dataset>
<identifier> ::= "a string"
// character '/' should be used with care.
<pos_list> ::= <pos_int>, <pos_list> | <pos_int>
<int_value> ::= 0 | <pos_int>
<pos_int> ::= [1-9][0-9]*
<hex_value> ::= 0x[0-F][0-F]+ | [0-F][0-F]+
\endcode
\section example110 An Example of an HDF5 File in DDL
\code{.unparsed}
HDF5 "example.h5" {
GROUP "/" {
ATTRIBUTE "attr1" {
DATATYPE H5T_STRING {
STRSIZE 17;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
"string attribute"
}
}
DATASET "dset1" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 10, 10 ) / ( 10, 10 ) }
DATA {
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
}
}
DATASET "dset2" {
DATATYPE H5T_COMPOUND {
H5T_STD_I32BE "a";
H5T_IEEE_F32BE "b";
H5T_IEEE_F64BE "c";
}
DATASPACE SIMPLE { ( 5 ) / ( 5 ) }
DATA {
{
1,
0.1,
0.01
},
{
2,
0.2,
0.02
},
{
3,
0.3,
0.03
},
{
4,
0.4,
0.04
},
{
5,
0.5,
0.05
}
}
}
GROUP "group1" {
COMMENT "This is a comment for group1";
DATASET "dset3" {
DATATYPE "/type1"
DATASPACE SIMPLE { ( 5 ) / ( 5 ) }
DATA {
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
}
}
}
}
DATASET "dset3" {
DATATYPE H5T_VLEN { H5T_STD_I32LE }
DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
DATA {
(0), (10, 11), (20, 21, 22), (30, 31, 32, 33)
}
}
GROUP "group2" {
HARDLINK "/group1"
}
SOFTLINK "slink1" {
LINKTARGET "somevalue"
}
DATATYPE "type1" H5T_COMPOUND {
H5T_ARRAY { [4] H5T_STD_I32BE } "a";
H5T_ARRAY { [5][6] H5T_IEEE_F32BE } "b";
}
}
}
\endcode
*/
doxygen/dox/DDLBNF112.dox
+653
View File
@@ -0,0 +1,653 @@
/** \page DDLBNF112 DDL in BNF for HDF5 1.12 and above
\todo Revise this & break it up!
\section intro112 Introduction
This document contains the data description language (DDL) for an HDF5 file. The
description is in Backus-Naur Form (BNF).
\section expo112 Explanation of Symbols
This section contains a brief explanation of the symbols used in the DDL.
\code{.unparsed}
::= defined as
<tname> a token with the name tname
<a> | <b> one of <a> or <b>
<a>opt zero or one occurrence of <a>
<a>* zero or more occurrence of <a>
<a>+ one or more occurrence of <a>
[0-9] an element in the range between 0 and 9
'[' the token within the quotes (used for special characters)
TBD To Be Decided
\endcode
\section ddl112 The DDL
\code{.unparsed}
<file> ::= HDF5 <file_name> { <file_super_block>opt <root_group> }
<file_name> ::= <identifier>
<file_super_block> ::= SUPER_BLOCK {
SUPERBLOCK_VERSION <int_value>
FREELIST_VERSION <int_value>
SYMBOLTABLE_VERSION <int_value>
OBJECTHEADER_VERSION <int_value>
OFFSET_SIZE <int_value>
LENGTH_SIZE <int_value>
BTREE_RANK <int_value>
BTREE_LEAF <int_value>
ISTORE_K <int_value>
<super_block_filespace>
USER_BLOCK {
USERBLOCK_SIZE <int_value>
}
}
<super_block_filespace> ::= FILE_SPACE_STRATEGY <super_block_strategy>
FREE_SPACE_PERSIST <boolean_value>
FREE_SPACE_SECTION_THRESHOLD <int_value>
FILE_SPACE_PAGE_SIZE <int_value>
<super_block_strategy> ::= H5F_FSPACE_STRATEGY_FSM_AGGR | H5F_FSPACE_STRATEGY_PAGE |
H5F_FSPACE_STRATEGY_AGGR | H5F_FSPACE_STRATEGY_NONE |
Unknown strategy
<root_group> ::= GROUP "/" {
<anon_named_datatype>*
<object_id>opt
<group_comment>opt
<group_attribute>*
<group_member>*
}
<datatype> ::= <atomic_type> | <compound_type> | <variable_length_type> | <array_type>
<anon_named_datatype> ::= DATATYPE <anon_named_type_name> {
<datatype>
}
<anon_named_type_name> ::= the assigned name for anonymous named type is
in the form of #oid, where oid is the object id
of the type
<atomic_type> ::= <integer> | <float> | <time> | <string> |
<bitfield> | <opaque> | <reference> | <enum>
<boolean_value> ::= FALSE | TRUE
<integer> ::= H5T_STD_I8BE | H5T_STD_I8LE |
H5T_STD_I16BE | H5T_STD_I16LE |
H5T_STD_I32BE | H5T_STD_I32LE |
H5T_STD_I64BE | H5T_STD_I64LE |
H5T_STD_U8BE | H5T_STD_U8LE |
H5T_STD_U16BE | H5T_STD_U16LE |
H5T_STD_U32BE | H5T_STD_U32LE |
H5T_STD_U64BE | H5T_STD_U64LE |
H5T_NATIVE_CHAR | H5T_NATIVE_UCHAR |
H5T_NATIVE_SHORT | H5T_NATIVE_USHORT |
H5T_NATIVE_INT | H5T_NATIVE_UINT |
H5T_NATIVE_LONG | H5T_NATIVE_ULONG |
H5T_NATIVE_LLONG | H5T_NATIVE_ULLONG
<float> ::= H5T_IEEE_F32BE | H5T_IEEE_F32LE |
H5T_IEEE_F64BE | H5T_IEEE_F64LE |
H5T_NATIVE_FLOAT | H5T_NATIVE_DOUBLE |
H5T_NATIVE_LDOUBLE
<time> ::= H5T_TIME: not yet implemented
<string> ::= H5T_STRING {
STRSIZE <strsize>;
STRPAD <strpad>;
CSET <cset>;
CTYPE <ctype>;
}
<strsize> ::= <int_value>
<strpad> ::= H5T_STR_NULLTERM | H5T_STR_NULLPAD | H5T_STR_SPACEPAD
<cset> ::= H5T_CSET_ASCII | H5T_CSET_UTF8
<ctype> ::= H5T_C_S1 | H5T_FORTRAN_S1
<bitfield> ::= H5T_STD_B8BE | H5T_STD_B8LE |
H5T_STD_B16BE | H5T_STD_B16LE |
H5T_STD_B32BE | H5T_STD_B32LE |
H5T_STD_B64BE | H5T_STD_B64LE
<opaque> ::= H5T_OPAQUE {
OPAQUE_TAG <identifier>;
OPAQUE_SIZE <int_value>;opt
}
<reference> ::= H5T_REFERENCE { <ref_type> }
<ref_type> ::= H5T_STD_REF_OBJECT | H5T_STD_REF_DSETREG | H5T_STD_REF | UNDEFINED
<compound_type> ::= H5T_COMPOUND {
<member_type_def>+
}
<member_type_def> ::= <datatype> <field_name>;
<field_name> ::= <identifier>
<variable_length_type> ::= H5T_VLEN { <datatype> }
<array_type> ::= H5T_ARRAY { <dim_sizes> <datatype> }
<dim_sizes> ::= '['<dimsize>']' | '['<dimsize>']'<dim_sizes>
<dimsize> ::= <int_value>
<attribute> ::= ATTRIBUTE <attr_name> {
<dataset_type>
<dataset_space>
<data>opt
}
<attr_name> ::= <identifier>
<dataset_type> ::= DATATYPE <path_name> | <datatype>
<enum> ::= H5T_ENUM {
<enum_base_type> <enum_def>+
}
<enum_base_type> ::= <integer>
// Currently enums can only hold integer type data, but they may be expanded
// in the future to hold any datatype
<enum_def> ::= <enum_symbol> <enum_val>;
<enum_symbol> ::= <identifier>
<enum_val> ::= <int_value>
<path_name> ::= <path_part>+
<path_part> ::= /<identifier>
<dataspace> ::= <scalar_space> | <simple_space> | <complex_space> | <null_space>
<null_space> ::= NULL
<scalar_space> ::= SCALAR
<simple_space> ::= SIMPLE { <current_dims> / <max_dims> }
<complex_space> ::= COMPLEX { <complex_space_definition> }
<dataset_space> ::= DATASPACE <path_name> | <dataspace>
<current_dims> ::= <dims>
<max_dims> ::= '(' <max_dim_list> ')'
<max_dim_list> ::= <max_dim> | <max_dim>, <max_dim_list>
<max_dim> ::= <int_value> | H5S_UNLIMITED
<data> ::= <subset> | <data_values>
<data_values> ::= DATA {
<scalar_space_data> | <simple_space_data>
}
<scalar_space_data> ::= <any_element>
<any_element> ::= <atomic_element> | <compound_element> |
<variable_length_element> | <array_element>
<any_data_seq> ::= <any_element> | <any_element>, <any_data_seq>
<atomic_element> :: = <integer_data> | <float_data> | <time_data> |
<string_data> | <bitfield_data> | <opaque_data> |
<enum_data> | <reference_data>
<subset> ::= SUBSET {
<start>;
<stride>;
<count>;
<block>;
DATA {
<simple_space_data>
}
}
<start> ::= START (<coor_list>)
<stride> ::= STRIDE (<pos_list>)
<count> ::= COUNT (<max_dim_list>)
<block> ::= BLOCK (<max_dim_list>)
<coor_list> ::= <coor_data>, <coor_list> | <coor_data>
<coor_data> ::= <integer_data> | H5S_UNLIMITED
<integer_data> ::= <int_value>
<float_data> ::= a floating point number
<time_data> ::= DATA{ not yet implemented.}
<string_data> ::= a string
// A string is enclosed in double quotes.
// If a string is displayed on more than one line, string concatenate
// operator '//'is used.
<bitfield_data> ::= <hex_value>
<opaque_data> ::= <hex_value>:<hex_value> | <hex_value>
<enum_data> ::= <enum_symbol>
<reference_data> ::= <object_ref_data> | <data_region_data> | <attribute_data> | NULL
<object_ref_data> ::= <object_type> <object_ref>
<object_type> ::= ATTRIBUTE | DATASET | GROUP | DATATYPE
<object_ref> ::= <object_id>
<object_id> ::= <path_name> | OBJECTID { <object_num> }
<object_num> ::= <int_value>:<int_value> | <int_value>
<attribute_data> ::= ATTRIBUTE <attr_name>opt
<data>opt
<data_region_data> ::= DATASET <dataset_name> {
<data_region_type>opt <data_region_data_list>
<dataset_type>opt <dataset_space>opt
<data>opt
}
<data_region_type> ::= REGION_TYPE <data_region_data_type>
<data_region_data_type> ::= POINT | BLOCK
<data_region_data_list> ::= <data_region_data_info>, <data_region_data_list> |
<data_region_data_info>
<data_region_data_info> ::= <region_info> | <point_info>
<region_info> ::= (<lower_region_vals>)-(<upper_region_vals>)
<lower_region_vals> ::= <lower_bound>, <lower_region_vals> | <lower_bound>
<upper_region_vals> ::= <upper_bound>, <upper_region_vals> | <upper_bound>
<lower_bound> ::= <int_value>
<upper_bound> ::= <int_value>
<point_info> ::= (<point_vals>)
<point_vals> ::= <int_value> | <int_value>, <point_vals>
<compound_element> ::= { <any_data_seq> }
<atomic_simple_data> :: = <atomic_element>, <atomic_simple_data> |
<atomic_element>
<simple_space_data> :: = <any_data_seq>
<variable_length_element> ::= ( <any_data_seq> )
<array_element> ::= '[' <any_data_seq> ']'
<named_datatype> ::= DATATYPE <type_name> { <datatype> }
<type_name> ::= <identifier>
<hardlink> ::= HARDLINK <path_name>
<group> ::= GROUP <group_name> { <hardlink> | <group_info> }
<group_comment> ::= COMMENT <string_data>
<group_name> ::= <identifier>
<group_info> ::= <object_id>opt <group_comment>opt <group_attribute>*
<group_member>*
<group_attribute> ::= <attribute>
<group_member> ::= <named_datatype> | <group> | <dataset> |
<softlink> | <external_link>
<dataset> ::= DATASET <dataset_name> { <hardlink> | <dataset_info> }
<dataset_info> ::= <dataset_type>
<dataset_space>
<dcpl_info>opt
<dataset_attribute>* <object_id>opt
<data>opt
// Tokens above can be in any order as long as <data> is
// after <dataset_type> and <dataset_space>.
<dcpl_info> ::= <storagelayout>
<compression_filters>
<fillvalue>
<allocationtime>
<dataset_name> ::= <identifier>
<storagelayout> :: = STORAGE_LAYOUT {
<contiguous_layout> | <chunked_layout> |
<compact_layout> | <virtual_layout>
}
<contiguous_layout> ::= CONTIGUOUS
<internal_layout> | <external_layout>
<chunked_layout> ::= CHUNKED <dims>
<filter_ratio>opt
<compact_layout> ::= COMPACT
<size>
<internal_layout> ::= <size>
<offset>
<external_layout> ::= EXTERNAL {
<external_file>+
}
<virtual_layout> ::= <vmaps>*opt
<vmaps> ::= MAPPING <int_value> {
<virtual_map>
<source_map>
}
<virtual_map> ::= VIRTUAL {
<vmaps_selection>
}
<source_map> ::= SOURCE {
FILE <file_name>
DATASET <dataset_name>
<vmaps_selection>
}
<vmaps_selection> ::= <regular_hyperslab> | <irregular_hyperslab> |
<select_points> | <select_none> | <select_all>
<regular_hyperslab> ::= SELECTION REGULAR_HYPERSLAB {
<start>
<stride>
<count>
<block>
}
<irregular_hyperslab> ::= SELECTION IRREGULAR_HYPERSLAB {
<region_info>+
}
<select_points> ::= SELECTION POINT {
(<coor_list>)+
}
<select_none> ::= SELECTION NONE
<select_all> ::= SELECTION ALL
<dims> ::= (<dims_values>)
<dims_values> ::= <int_value> | <int_value>, <dims_values>
<external_file> ::= FILENAME <file_name> <size> <offset>
<offset> ::= OFFSET <int_value>
<size> ::= SIZE <int_value>
<filter_ratio> ::= <size> | <compressionratio>
<compressionratio> :: = <size> (<float_data>:1 COMPRESSION)
<compression_filters> :: = FILTERS {
<filter_type>+ | NONE
}
<filter_type> :: = <filter_deflate> | <filter_shuffle> |
<filter_flecther> | <filter_szip> |
<filter_nbit> | <filter_scaleoffset> |
<filter_default>
<filter_default> :: = <filter_user> {
FILTER_ID <int_value>
<filter_comment>opt
<filter_params>opt
}
<filter_user> :: = USER_DEFINED_FILTER
<filter_deflate> :: = COMPRESSION DEFLATE { LEVEL <int_value> }
<filter_shuffle> :: = PREPROCESSING SHUFFLE
<filter_flecther> :: = CHECKSUM FLETCHER32
<filter_szip> :: = COMPRESSION SZIP {
PIXELS_PER_BLOCK <int_value>
<filter_szip_mode>opt
<filter_szip_coding>opt
<filter_szip_order>opt
<filter_szip_header>opt
}
<filter_szip_mode> :: = MODE HARDWARE | K13
<filter_szip_coding> :: = CODING ENTROPY | NEAREST NEIGHBOUR
<filter_szip_order> :: = BYTE_ORDER LSB | MSB
<filter_szip_header> :: = HEADER RAW
<filter_nbit> :: = CHECKSUM NBIT
<filter_scaleoffset> :: = COMPRESSION SCALEOFFSET { MIN BITS <int_value> }
<filter_comment> :: = COMMENT <identifier>
<filter_params> :: = PARAMS { <int_value>* }
<fillvalue> ::= FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC | H5D_FILL_TIME_NEVER | H5D_FILL_TIME_IFSET
VALUE H5D_FILL_VALUE_UNDEFINED | H5D_FILL_VALUE_DEFAULT | <any_element>
}
<allocationtime> ::= ALLOCATION_TIME {
H5D_ALLOC_TIME_EARLY | H5D_ALLOC_TIME_INCR |
H5D_ALLOC_TIME_LATE
}
<dataset_attribute> ::= <attribute>
<softlink> ::= SOFTLINK <softlink_name> {
LINKTARGET <target>
}
<softlink_name> ::= <identifier>
<target> ::= <identifier>
<external_link> ::= EXTERNAL_LINK <external_link_name> {
TARGETFILE <targetfile>
TARGETPATH <targetpath> <targetobj>opt
}
<external_link_name> ::= <identifier>
<user_defined_link> ::= USERDEFINED_LINK <external_link_name> {
LINKCLASS <user_link_type>
}
<user_link_type> ::= <int_value>
<targetfile> ::= <file_name>
<targetpath> ::= <identifier>
<targetobj> ::= <named_datatype> | <group> | <dataset>
<identifier> ::= "a string"
// character '/' should be used with care.
<pos_list> ::= <pos_int>, <pos_list> | <pos_int>
<int_value> ::= 0 | <pos_int>
<pos_int> ::= [1-9][0-9]*
<hex_value> ::= 0x[0-F][0-F]+ | [0-F][0-F]+
\endcode
\section example112 An Example of an HDF5 File in DDL
\code{.unparsed}
HDF5 "example.h5" {
GROUP "/" {
ATTRIBUTE "attr1" {
DATATYPE H5T_STRING {
STRSIZE 17;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
DATA {
"string attribute"
}
}
DATASET "dset1" {
DATATYPE H5T_STD_I32BE
DATASPACE SIMPLE { ( 10, 10 ) / ( 10, 10 ) }
DATA {
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
}
}
DATASET "dset2" {
DATATYPE H5T_COMPOUND {
H5T_STD_I32BE "a";
H5T_IEEE_F32BE "b";
H5T_IEEE_F64BE "c";
}
DATASPACE SIMPLE { ( 5 ) / ( 5 ) }
DATA {
{
1,
0.1,
0.01
},
{
2,
0.2,
0.02
},
{
3,
0.3,
0.03
},
{
4,
0.4,
0.04
},
{
5,
0.5,
0.05
}
}
}
GROUP "group1" {
COMMENT "This is a comment for group1";
DATASET "dset3" {
DATATYPE "/type1"
DATASPACE SIMPLE { ( 5 ) / ( 5 ) }
DATA {
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
},
{
[ 0, 1, 2, 3 ],
[ 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.5, 0.5, 0.5, 0.5, 0.5, 0.5 ]
}
}
}
}
DATASET "dset3" {
DATATYPE H5T_VLEN { H5T_STD_I32LE }
DATASPACE SIMPLE { ( 4 ) / ( 4 ) }
DATA {
(0), (10, 11), (20, 21, 22), (30, 31, 32, 33)
}
}
GROUP "group2" {
HARDLINK "/group1"
}
SOFTLINK "slink1" {
LINKTARGET "somevalue"
}
DATATYPE "type1" H5T_COMPOUND {
H5T_ARRAY { [4] H5T_STD_I32BE } "a";
H5T_ARRAY { [5][6] H5T_IEEE_F32BE } "b";
}
}
}
\endcode
*/
doxygen/dox/FileFormatSpec.dox
+23
View File
@@ -0,0 +1,23 @@
/** \page FMT3 HDF5 File Format Specification Version 3.0
\htmlinclude H5.format.html
*/
/** \page FMT2 HDF5 File Format Specification Version 2.0
\htmlinclude H5.format.2.0.html
*/
/** \page FMT11 HDF5 File Format Specification Version 1.1
\htmlinclude H5.format.1.1.html
*/
/** \page FMT1 HDF5 File Format Specification Version 1.0
\htmlinclude H5.format.1.0.html
*/
doxygen/dox/GettingStarted.dox
+3
View File
@@ -0,0 +1,3 @@
/** \page GettingStarted \Code{Hello, HDF5!}
*/
doxygen/dox/H5Fget_info.dox
+2 -5
View File
@@ -6,9 +6,8 @@
* Similarly, the macro for the \ref H5F_info_t struct is mapped to either
* H5F_info1_t or H5F_info2_t.
*
* Such macros are provided to facilitate application
* compatibility. Their use and mappings are fully described in "API Compatibility
* Macros in HDF5".
* Such macros are provided to facilitate application compatibility.
* Their use and mappings are fully described in \ref api-compat-macros.
*
* When both the HDF5 library and the application are built and installed with
* no specific compatibility flags, H5Fget_info() is mapped to the most recent
@@ -37,8 +36,6 @@
* \li \Code{H5F_info_t_vers=2}: H5F_info2_t
* \li \Code{H5F_info_t_vers=1}: H5F_info1_t
*
* \todo Fix the reference.
*
* \version 1.10.0 The C function H5Fget_info() and H5F_info_t renamed to
* H5Fget_info1() and H5F_info1_t, respectively, and deprecated
* in this release. The C macro #H5Fget_info, the C function
doxygen/dox/H5Lget_info.dox
+2 -3
View File
@@ -3,8 +3,8 @@
* \def H5Lget_info()
* H5Lget_info() is a macro that is mapped to either H5Lget_info1()
* or H5Lget_info2() Such macros are provided to facilitate application
* compatibility. Their use and mappings are fully described in Compatibility
* Macros in HDF5. If the library and/or application is compiled for Release
* compatibility. Their use and mappings are fully described in \ref api-compat-macros.
* If the library and/or application is compiled for Release
* 1.12 emulation, H5Lget_info() will be mapped to H5Lget_info2() and
* H5Lget_info1() is deprecated. With earlier versions, H5Lget_info() is mapped to
* H5Lget_info1(). Specific compile-time compatibility flags and the resulting
@@ -14,5 +14,4 @@
* \li Emulate Release 1.12: H5Lget_info2()
* \li Emulate Release 1.8 or 1.10 interface: H5Lget_info1()
*
* \todo Fix the reference.
*/
doxygen/dox/H5Lget_info_by_idx.dox
+2 -3
View File
@@ -3,8 +3,8 @@
* \def H5Lget_info_by_idx()
* H5Lget_info_by_idx() is a macro that is mapped to either H5Lget_info_by_idx1()
* or H5Lget_info_by_idx2() Such macros are provided to facilitate application
* compatibility. Their use and mappings are fully described in Compatibility
* Macros in HDF5. If the library and/or application is compiled for Release
* compatibility. Their use and mappings are fully described in \ref api-compat-macros.
* If the library and/or application is compiled for Release
* 1.12 emulation, H5Lget_info_by_idx() will be mapped to H5Lget_info_by_idx2() and
* H5Lget_info_by_idx1() is deprecated. With earlier versions, H5Lget_infoby_idx() is mapped to
* H5Lget_info_by_idx1(). Specific compile-time compatibility flags and the resulting
@@ -14,5 +14,4 @@
* \li Emulate Release 1.12: H5Lget_info_by_idx2()
* \li Emulate Release 1.8 or 1.10 interface: H5Lget_info_by_idx1()
*
* \todo Fix the reference.
*/
doxygen/dox/H5Literate.dox
+1 -3
View File
@@ -4,7 +4,7 @@
* H5Literate() is a macro that is mapped to either H5Literate1() or
* H5Literate2() Such macros are provided to facilitate application
* compatibility. Their use and mappings are fully described in
* Compatibility Macros in HDF5. If the library and/or application is
* \ref api-compat-macros. If the library and/or application is
* compiled for Release 1.12 emulation, H5Literate() will be mapped to
* H5Literate2() and H5Literate1() is deprecated. With earlier versions,
* H5Literate() is mapped to H5Literate1(). Specific compile-time compatibility
@@ -14,8 +14,6 @@
* \li Emulate Release 1.12: H5Literate2()
* \li Emulate Release 1.8 or 1.10 interface: H5Literate1()
*
* \todo Fix the reference.
*
* \version 1.12.0 The function H5Literate() was renamed to H5Literate1() and
* deprecated in this release. The macro H5Literate() and the
* function H5Literate2() were introduced in this release.
doxygen/dox/H5Literate_by_name.dox
+1 -3
View File
@@ -4,7 +4,7 @@
* H5Literate_by_name() is a macro that is mapped to either
* H5Literate_by_name1() or H5Literate_by_name2() Such macros are provided to
* facilitate application compatibility. Their use and mappings are fully
* described in Compatibility Macros in HDF5. If the library and/or application is
* described in \ref api-compat-macros. If the library and/or application is
* compiled for Release 1.12 emulation, H5Literate_by_name() will be mapped to
* H5Literate_by_name2() and H5Literate_by_name1() is deprecated. With earlier
* versions, H5Literate_by_name() is mapped to H5Literate_by_name1().
@@ -15,8 +15,6 @@
* \li Emulate Release 1.12: H5Literate_by_name2()
* \li Emulate Release 1.8 or 1.10 interface: H5Literate_by_name1()
*
* \todo Fix the reference.
*
* \version 1.12.0 The function H5Literate_by_name() was renamed to H5Literate_by_name1() and
* deprecated in this release. The macro H5Literate_by_name() and the
* function H5Literate_by_name2() were introduced in this release.
doxygen/dox/H5Lvisit.dox
+1 -3
View File
@@ -4,7 +4,7 @@
* H5Lvisit() is a macro that is mapped to either H5Lvisit1() or
* H5Lvisit2() Such macros are provided to facilitate application
* compatibility. Their use and mappings are fully described in
* Compatibility Macros in HDF5. If the library and/or application is
* \ref api-compat-macros. If the library and/or application is
* compiled for Release 1.12 emulation, H5Lvisit() will be mapped to
* H5Lvisit2() and H5Lvisit1() is deprecated. With earlier versions,
* H5Lvisit() is mapped to H5Lvisit1(). Specific compile-time compatibility
@@ -14,8 +14,6 @@
* \li Emulate Release 1.12: H5Lvisit2()
* \li Emulate Release 1.8 or 1.10 interface: H5Lvisit1()
*
* \todo Fix the reference.
*
* \version 1.12.0 The function H5Lvisit() was renamed to H5Lvisit1() and
* deprecated in this release. The macro H5Lvisit() and the
* function H5Lvisit2() were introduced in this release.
doxygen/dox/H5Lvisit_by_name.dox
+1 -3
View File
@@ -4,7 +4,7 @@
* H5Lvisit_by_name() is a macro that is mapped to either H5Lvisit_by_name1() or
* H5Lvisit_by_name2() Such macros are provided to facilitate application
* compatibility. Their use and mappings are fully described in
* Compatibility Macros in HDF5. If the library and/or application is
* \ref api-compat-macros. If the library and/or application is
* compiled for Release 1.12 emulation, H5Lvisit_by_name() will be mapped to
* H5Lvisit_by_name2() and H5Lvisit_by_name1() is deprecated. With earlier versions,
* H5Lvisit_by_name() is mapped to H5Lvisit_by_name1(). Specific compile-time
@@ -14,8 +14,6 @@
* \li Emulate Release 1.12: H5Lvisit_by_name2()
* \li Emulate Release 1.8 or 1.10 interface: H5Lvisit_by_name1()
*
* \todo Fix the reference.
*
* \version 1.12.0 The function H5Lvisit_by_name() was renamed to H5Lvisit_by_name1() and
* deprecated in this release. The macro H5Lvisit_by_name() and the
* function H5Lvisit_by_name2() were introduced in this release.
doxygen/dox/MetadataCachingInHDF5.dox
+1020
View File
@@ -0,0 +1,1020 @@
/** \page TNMDC Metadata Caching in HDF5
\todo Revise this!
\section intro Introduction
In the 1.6.4 release, we introduced a re-implementation of the metadata
cache. That release contained an incomplete version of the cache which could not
be controlled via the API. The version in the 1.8 release is more mature and
includes new API calls that allow the user program to configure the metadata
cache both on file open and at run time.
From the user perspective, the most striking effect of the new cache should be a
large reduction in the cache memory requirements when working with complex HDF5
files.
Those working with such files may also notice a reduction in file close time.
Those working with HDF5 files with a simple structure shouldn't notice any
particular changes in most cases. In rare cases, there may be a significant
improvement in performance.
The remainder of this document contains an architectural overview of the old and
new metadata caches, a discussion of algorithms used to automatically adjust
cache size to circumstances, and a high-level discussion of the cache
configuration controls. It can be safely skipped by anyone who works only with
HDF5 files with relatively simple structure (i.e. no huge groups, no datasets
with large numbers of chunks, and no objects with large numbers of attributes.)
On the other hand, it is mandatory reading if you want to use something other
than the default metadata cache configuration. The documentation on the metadata
cache-related API calls will not make much sense without this background.
\section oldnew Old and New Metadata Cache
\subsection old The Old Metadata Cache
The old metadata cache indexed the cache with a hash table with no provision for
collisions. Instead, collisions were handled by evicting the existing entry to
make room for the new entry. Aside from flushes, there was no other mechanism
for evicting entries, so the replacement policy could best be described as
"Evict on Collision".
As a result, if two frequently used entries hashed to the same location, they
would evict each other regularly. To decrease the likelihood of this situation,
the default hash table size was set fairly large -- slightly more than
10,000. This worked well, but since the size of metadata entries is not bounded,
and since entries were only evicted on collision, the large hash table size
allowed the cache size to explode when working with HDF5 files with complex
structure.
The "Evict on Collision" replacement policy also caused problems with the
parallel version of the HDF5 library, as a collision with a dirty entry could
force a write in response to a metadata read. Since all metadata writes must be
collective in the parallel case while reads need not be, this could cause the
library to hang if only some of the processes participated in a metadata read
that forced a write. Prior to the implementation of the new metadata cache, we
dealt with this issue by maintaining a shadow cache for dirty entries evicted by
a read.
\subsection new The New Metadata Cache
The new metadata cache was designed to address the above issues. After
implementation, it became evident that the working set size for HDF5 files
varies widely depending on both structure and access patterns. Thus it was
necessary to add support for cache size adjustment under either automatic or
user program control (see section 2.3 for details).
When the cache is operating under direct user program control, it is also
possible to temporarily disable evictions from the metadata cache so as to
maximize raw data throughput at the expense of allowing the cache to grow
without bound until evictions are enabled again.
Structurally, the new metadata cache can be thought of as a heavily modified
version of the UNIX buffer cache as described in chapter three of M. J. Bach's
"The Design of the UNIX Operating System" In essence, the UNIX buffer cache uses
a hash table with chaining to index a pool of fixed-size buffers. It uses the
LRU replacement policy to select candidates for eviction.
Since HDF5 metadata entries are not of fixed size and may grow arbitrarily
large, the size of the new metadata cache cannot be controlled by setting a
maximum number of entries. Instead, the new cache keeps a running sum of the
sizes of all entries and attempts to evict entries as necessary to stay within a
user-specified maximum size. (Note the use of the word "attempts" here -- as
will be seen, it is possible for the cache to exceed its currently specified
maximum size.) At present, the LRU replacement policy is the only option for
selecting candidates for eviction.
Per the standard Unix buffer cache, dirty entries are given two passes through
the LRU list before being evicted. The first time they reach the end of the LRU
list, they are flushed, marked as clean, and moved to the head of the LRU
list. When a clean entry reaches the end of the LRU list, it is simply evicted
if space is needed.
The cache cannot evict entries that are locked, and thus it will temporarily
grow beyond its maximum size if there are insufficient unlocked entries
available for eviction.
In the parallel version of the library, only the cache running under process 0
of the file communicator is allowed to write metadata to file. All the other
caches must retain dirty metadata until the process 0 cache tells them that the
metadata is clean.
Since all operations modifying metadata must be collective, all caches see the
same stream of dirty metadata. This fact is used to allow them to synchronize
every n bytes of dirty metadata, where n is a user-configurable value that
defaults to 256 KB.
To avoid sending the other caches messages from the future, process 0 must not
write any dirty entries until it reaches a synchronization point. When it
reaches a synchronization point, it writes entries as needed, and then
broadcasts the list of flushed entries to the other caches. The caches on the
other processes use this list to mark entries clean before they leave the
synchronization point, allowing them to evict those entries as needed.
The caches will also synchronize on a user-initiated flush.
To minimize overhead when running in parallel, the cache maintains a "clean" LRU
list in addition to the regular LRU list. This list contains only clean entries
and is used as a source of candidates for eviction when flushing dirty entries
is not allowed.
Since flushing entries is forbidden most of the time when running in parallel,
the caches can be forced to exceed their maximum sizes if they run out of clean
entries to evict.
To decrease the likelihood of this event, the new cache allows the user to
specify a minimum clean size -- which is a minimum total size of all the entries
on the clean LRU plus all unused space in the cache.
While the clean LRU list is only maintained in the parallel version of the HDF5
library, the notion of a minimum clean size still applies in the serial
case. Here it is used to force a mix of clean and dirty entries in the cache
even in the write-only case.
This, in turn, reduces the number of redundant flushes by avoiding the case in
which the cache fills with dirty metadata and all entries must be flushed before
a clean entry can be evicted to make room for a new entry.
Observe that in both the serial and parallel cases, the maintenance of a minimum
clean size modifies the replacement policy, as dirty entries may be flushed
earlier than would otherwise be the case so as to maintain the desired amount of
clean and/or empty space in the cache.
While the new metadata cache only supports the LRU replacement policy at
present, that may change. Support for multiple replacement policies was very
much in mind when the cache was designed, as was the ability to switch
replacement policies at run time. The situation has been complicated by the
later addition of the adaptive cache resizing requirement, as two of the
resizing algorithms piggyback on the LRU list. However, if there is a need for
additional replacement policies, it shouldn't be too hard to implement them.
\section adapt Adaptive Cache Resizing in the New Metadata Cache
As mentioned earlier, the metadata working set size for an HDF5 file varies
wildly depending on the structure of the file and the access pattern. For
example, a 2MB limit on metadata cache size is excessive for an H5repack of
almost all HDF5 files we have tested. However, I have a file submitted by one of
our users that will run a 13% hit rate with this cache size and will lock up one
of our Linux boxes using the old metadata cache. Increase the new metadata cache
size to 4 MB, and the hit rate exceeds 99%.
In this case, the main culprit is a root group with more than 20,000 entries in
it. As a result, the root group heap exceeds 1 MB, which tends to crowd out the
rest of the metadata in a 2 MB cache
This case and a number of synthetic tests convinced us that we needed to modify
the new metadata cache to expand and contract according to need within
user-specified bounds.
I was unable to find any previous work on this problem, so I invented solutions
as I went along. If you are aware of prior work, please send me references. The
closest I was able to come was a group of embedded CPU designers who were
turning off sections of their cache to conserve power.
\subsection increasing Increasing the Cache Size
In the context of the HDF5 library, the problem of increasing the cache size as
necessary to contain the current working set turns out to involve two rather
different issues.
The first of these, which was recognized immediately, is the problem of
recognizing long term changes in working set size, and increasing the cache size
accordingly, while not reacting to transients.
The second, which I recognized the hard way, is to adjust the cache size for
sudden, dramatic increases in working set size caused by requests for large
pieces of metadata which may be larger than the current metadata cache size.
The algorithms for handling these situations are discussed below. These problems
are largely orthogonal to each other, so both algorithms may be used
simultaneously.
\subsubsection hrtcsi Hit Rate Threshold Cache Size Increment
Perhaps the most obvious heuristic for identifying cases in which the cache is
too small involves monitoring the hit rate. If the hit rate is low for a while,
and the cache is at its current maximum size, the current maximum cache size is
probably too small.
The hit rate threshold algorithm for increasing cache size applies this
intuition directly.
Hit rate statistics are collected over a user-specified number of cache
accesses. This period is known as an epoch.
At the end of each epoch, the hit rate is computed, and the counters are
reset. If the hit rate is below a user-specified threshold and the cache is at
its current maximum size, the maximum size of the cache is increased by a
user-specified multiple. If required, the new cache maximum size is clipped to
stay within the user-specified upper bound on the maximum cache size, and
optionally, within a user-specified maximum increment.
My tests indicate that this algorithm works well in most cases. However, in a
synthetic test in which hit rate increased slowly with cache size, and load
remained steady for many epochs, I observed a case in which cache size increased
until the hit rate just exceeded the specified minimum and then stalled. This is
a problem, as to avoid volatility, it is necessary to set the minimum hit rate
threshold well below the desired hit rate. Thus we may find ourselves with a
cache running with a 91% hit rate when we really want it to increase its size
until the hit rate is about 99%.
If this case occurs frequently in actual use, I will have to come up with an
improved algorithm. Please let me know if you see this behavior. However, I had
to work rather hard to create it in my synthetic tests, so I would expect it to
be uncommon.
\subsubsection fcsi Flash Cache Size Increment
A fundamental problem with the above algorithm is that contains the hidden
assumption that cache entries are relatively small in comparison to the cache
itself. While I knew this assumption was not generally true when I developed the
algorithm, I thought that cases, where it failed, would be so rare as to not be
worth considering, as even if they did occur, the above algorithm would rectify
the situation within an epoch or two.
While it is true that such occurrences are rare, and it is true that the hit
rate threshold cache size increment algorithm will rectify the situation
eventually, the performance degradation experienced by users while waiting for
the epoch to end was so extreme that some way of accelerating response to such
situations was essential.
To understand the problem, consider the following use case:
Suppose we create a group, and then repeatedly create a new data set in the
group, write some data to it and then close it.
In some versions of the HDF5 file format, the names of the datasets will be
stored in a local heap associated with the group, and the space for that heap
will be allocated in a single, contiguous chunk. When this local heap is full,
we allocate a new chunk twice the size of the old, copy the data from the old
local heap into the new, and discard the old local heap.
By default, the minimum metadata cache size is set to 2 MB. Thus in this use
case, our hit rate will be fine as long as the local heap is no larger than a
little less than 2 MB, as the group related metadata is accessed frequently and
never evicted, and the data set related metadata is never accessed once the data
set is closed, and thus is evicted smoothly to make room for new data sets.
All this changes abruptly when the local heap finally doubles in size to a value
above the slightly less than 2 MB limit. All of a sudden, the local heap is the
size of the metadata cache, and the cache must constantly swap it in to access
it, and then swap it out to make room for other metadata.
The hit rate threshold-based algorithm for increasing the cache size will fix
this problem eventually, but performance will be very bad until it does, as the
metadata cache will largely ineffective until its size is increased.
An obvious heuristic for addressing this "big rock in a small pond" issue is to
watch for large "incoming rocks", and increase the size of the "pond" if the
rock is so big that it will force most of the "water" out of the "pond".
The add space flash cache size increment algorithm applies this intuition
directly:
Let x be either the size of a newly inserted entry, a newly loaded entry, or the
number of bytes by which the size of an existing entry has been increased
(i.e. the size of the "rock").
If x is greater than some user-specified fraction of the current maximum cache
size, increase the current maximum cache size by x times some user-specified
multiple, less any free space that was in the cache, to begin with. Further, to
avoid confusing the other cache size increment/decrement code, start a new
epoch.
At present, this algorithm pays no attention to any user-specified limit on the
maximum size of any single cache size increase, but it DOES stay within the
user-specified upper bound on the maximum cache size.
While it should be easy to see how this algorithm could be fooled into
inactivity by a large number of entries that were not quite large enough to
cross the threshold, in practice it seems to work reasonably well.
Needless to say, I will revisit the issue should this cease to be the case.
\subsection decreasing Decreasing the Cache Size
Identifying cases in which the maximum cache size is larger than necessary
turned out to be more difficult.
\subsubsection hrtcsr Hit Rate Threshold Cache Size Reduction
One obvious heuristic is to monitor the hit rate and guess that we can safely
decrease cache size if the hit rate exceeds some user-supplied threshold (say
.99995). The hit rate threshold size decrement algorithm implemented in the new
metadata cache implements this intuition as follows:
At the end of each epoch (this is the same epoch that is used in the cache size
increment algorithm), the hit rate is compared with the user-specified
threshold. If the hit rate exceeds that threshold, the current maximum cache
size is decreased by a user-specified factor. If required, the size of the
reduction is clipped to stay within a user-specified lower bound on the maximum
cache size, and optionally, within a user-specified maximum decrement.
In my synthetic tests, this algorithm works poorly. Even with a very high
threshold and a small maximum reduction, it results in cache size
oscillations. The size increment code typically increments the maximum cache
size above the working set size. This results in a high hit rate, which causes
the threshold size decrement code to reduce the maximum cache size below the
working set size, which causes the hit rate to crash causing the cycle to
repeat. The resulting average hit rate is poor.
It remains to be seen if this behavior will be seen in the field. The algorithm
is available for use, but it wouldn't be my first choice. If you use it, please
report back.
\subsubsection acsr Ageout Cache Size Reduction
Another heuristic for dealing with oversized cache conditions is to look for
entries that haven't been accessed for a long time, evict them, and reduce the
cache size accordingly.
The age out cache size reduction applies this intuition as follows: At the end
of each epoch (again the same epoch as used in the cache size increment
algorithm), all entries that haven't been accessed for a user-configurable
number of epochs (1 - 10 at present) are evicted. The maximum cache size is then
reduced to equal the sum of the sizes of the remaining entries. The size of the
reduction is clipped to stay within a user-specified lower bound on maximum
cache size, and optionally, within a user-specified maximum decrement.
In addition, the user may specify a minimum fraction of the cache which must be
empty before the cache size is reduced. Thus if an empty reserve of 0.1 was
specified on a 10 MB cache, there would be no cache size reduction unless the
eviction of aged out entries resulted in more than 1 MB of empty space. Further,
even after the reduction, the cache would be one-tenth empty.
In my synthetic tests, the age out algorithm works rather well, although it is
somewhat sensitive to the epoch length and age out period selection.
\subsubsection awhrtcsr Ageout With Hit Rate Threshold Cache Size Reduction
To address these issues, I combined the hit rate threshold and age out
heuristics.
Age out with threshold works just like age out, except that the algorithm is not
run unless the hit rate exceeded a user-specified threshold in the previous
epoch.
In my synthetic tests, age out with threshold seems to work nicely, with no
observed oscillation. Thus I have selected it as the default cache size
reduction algorithm.
For those interested in such things, the age out algorithm is implemented by
inserting a marker entry at the head of the LRU list at the beginning of each
epoch. Entries that haven't been accessed for at least n epochs are simply
entries that appear in the LRU list after the n-th marker at the end of an
epoch.
\section configuring Configuring the New Metadata Cache
Due to a lack of resources, the design work on the automatic cache size
adjustment algorithms was done hastily, using primarily synthetic tests. I don't
think I spent more than a couple weeks writing and running performance tests --
most time went into coding and functional testing.
As a result, while I think the algorithms provided for adaptive cache resizing
will work well in actual use, I don't really know (although preliminary results
from the field are promising). Fortunately, the issue shouldn't arise for the
vast majority of HDF5 users, and those for whom it may arise should be savvy
enough to recognize problems and deal with them.
For this latter class of users, I have implemented a number of new API calls
allowing the user to select and configure the cache resize algorithms, or to
turn them off and control cache size directly from the user program. There are
also API calls that allow the user program to monitor hit rate and cache size.
From the user perspective, all the cache configuration data for a given file is
contained in an instance of the \ref H5AC_cache_config_t structure -- the definition
of which is given below:
\snippet H5ACpublic.h H5AC_cache_config_t_snip
This structure is defined in \c H5ACpublic.h. Each field is discussed below and in
the associated header comment.
The C API allows you to get and set this structure directly. Unfortunately, the
Fortran API has to do this with individual parameters for each of the fields
(with the exception of version).
While the API calls are discussed individually in the reference manual, the
following high-level discussion of what fields to change for different purposes
should be useful.
\subsection gconfig General Configuration
The \c version field is intended to allow \THG to change the \c
H5AC_cache_config_t structure without breaking old code. For now, this field
should always be set to \c H5AC__CURR_CACHE_CONFIG_VERSION, even when you are
getting the current configuration data from the cache. The library needs the
version number to know where fields are located with reference to the supplied
base address.
The \ref H5AC_cache_config_t.rpt_fcn_enabled "rpt_fcn_enabled" field is a
boolean flag that allows you to turn on and off the resize reporting function
that reports the activities of the adaptive cache resize code at the end of each
epoch -- assuming that it is enabled.
The report function is unsupported, so you are on your own if you use it. Since
it dumps status data to stdout, you should not attempt to use it with Windows
unless you modify the source. You may find it useful if you want to experiment
with different adaptive resize configurations. It is also a convenient way of
diagnosing poor cache configuration. Finally, if you do lots of runs with
identical behavior, you can use it to determine the metadata cache size needed
in each phase of your program so you can set the required cache sizes manually.
The trace file fields are also unsupported. They allow one to open and close a
trace file in which all calls to the metadata cache are logged in a
user-specified file for later analysis. The feature is intended primarily for
THG use in debugging or optimizing the metadata cache in cases where users in
the field observe obscure failures or poor performance that we cannot re-create
in the lab. The trace file will allow us to re-create the exact sequence of
cache operations that are triggering the problem.
At present we do not have a playback utility for trace files, although I imagine
that we will write one quickly when and if we need it.
To enable the trace file, you load the full path of the desired trace file into
\ref H5AC_cache_config_t.trace_file_name "trace_file_name", and set \ref
H5AC_cache_config_t.open_trace_file "open_trace_file" to \c TRUE. In the
parallel case, an ASCII representation of the rank of each process is appended
to the supplied trace file name to create a unique trace file name for that
process.
To close an open trace file, set \ref H5AC_cache_config_t.close_trace_file
"close_trace_file" to \c TRUE.
It must be emphasized that you are on your own if you play with the trace file
feature absent a request from \THG. Needless to say, the trace file feature is
disabled by default. If you enable it, you will take a large performance hit and
generate huge trace files.
The \ref H5AC_cache_config_t.evictions_enabled "evictions_enabled" field is a
boolean flag allowing the user to disable the eviction of entries from the
metadata cache. Under normal operation conditions, this field will always be set
to \c TRUE.
In rare circumstances, the raw data throughput requirements may be so high that
the user wishes to postpone metadata writes so as to reserve I/O throughput for
raw data. The \ref H5AC_cache_config_t.evictions_enabled "evictions_enabled"
field exists to allow this -- although the user is to be warned that the
metadata cache will grow without bound while evictions are disabled. Thus
evictions should be re-enabled as soon as possible, and it may be wise to
monitor cache size and statistics (to see how to enable statistics, see the
debugging facilities section below).
Evictions may only be disabled when the automatic cache resize code is disabled
as well. Thus to disable evictions, not only must the user set the \ref
H5AC_cache_config_t.evictions_enabled "evictions_enabled" field to \c FALSE, but
he must also set \ref H5AC_cache_config_t.incr_mode "incr_mode" to
#H5C_incr__off, set \ref H5AC_cache_config_t.flash_incr_mode "flash_incr_mode"
to #H5C_flash_incr__off, and set \ref H5AC_cache_config_t.decr_mode "decr_mode"
to #H5C_decr__off.
To re-enable evictions, just set \ref H5AC_cache_config_t.evictions_enabled
"evictions_enabled" back to \c TRUE.
Before passing on to other subjects, it is worth re-iterating that disabling
evictions is an extreme step. Before attempting it, you might consider setting a
large cache size manually, and flushing the cache just before high raw data
throughput is required. This may yield the desired results without the risks
inherent in disabling evictions.
The \ref H5AC_cache_config_t.set_initial_size "set_initial_size" and \ref
H5AC_cache_config_t.initial_size "initial_size" fields allow you to specify an
initial maximum cache size. If \ref H5AC_cache_config_t.set_initial_size
"set_initial_size" is \c TRUE, \ref H5AC_cache_config_t.initial_size
"initial_size" must lie in the interval [\ref H5AC_cache_config_t.min_size
"min_size", \ref H5AC_cache_config_t.max_size "max_size"] (see below for a
discussion of the \ref H5AC_cache_config_t.min_size "min_size" and \ref
H5AC_cache_config_t.max_size "max_size" fields).
If you disable the adaptive cache resizing code (done by setting \ref
H5AC_cache_config_t.incr_mode "incr_mode" to #H5C_incr__off, \ref
H5AC_cache_config_t.flash_incr_mode "flash_incr_mode" to #H5C_flash_incr__off,
and \ref H5AC_cache_config_t.decr_mode "decr_mode" to #H5C_decr__off), you can
use these fields to control maximum cache size manually, as the maximum cache
size will remain at the initial size.
Note, that the maximum cache size is only modified when \ref
H5AC_cache_config_t.set_initial_size "set_initial_size" is \c TRUE. This allows
the use of configurations specified at compile time to change resize
configuration without altering the current maximum size of the cache. Without
this feature, an additional call would be required to get the current maximum
cache size so as to set the \ref H5AC_cache_config_t.initial_size "initial_size"
to the current maximum cache size, and thereby avoid changing it.
The \ref H5AC_cache_config_t.min_clean_fraction "min_clean_fraction" sets the
current minimum clean size as a fraction of the current max cache size. While
this field was originally used only in the parallel version of the library, it
now applies to the serial version as well. Its value must lie in the range
\Code{[0.0, 1.0]}. 0.01 is reasonable in the serial case, and 0.3 in the
parallel.
A potential interaction, discovered at release 1.8.3, between the enforcement of
the \ref H5AC_cache_config_t.min_clean_fraction "min_clean_fraction" and the
adaptive cache resize code can severely degrade performance. While this
interaction is easily dealt with in the serial case by setting \ref
H5AC_cache_config_t.min_clean_fraction "min_clean_fraction" to 0.01, the problem
is more difficult in the parallel case. Please see the Interactions section
below for further details.
The \ref H5AC_cache_config_t.max_size "max_size" and \ref
H5AC_cache_config_t.min_size "min_size" fields specify the range of maximum
sizes that may be set for the cache by the automatic resize code. \ref
H5AC_cache_config_t.min_size "min_size" must be less than or equal to
\ref H5AC_cache_config_t.max_size "max_size", and both must lie in the range
\Code{[H5C__MIN_MAX_CACHE_SIZE, H5C__MAX_MAX_CACHE_SIZE]} -- currently [1 KB,
128 MB]. If you routinely run a cache size in the top half of this range, you
should increase the hash table size. To do this, modify the \c
H5C__HASH_TABLE_LEN \Code{\#define} in \c H5Cpkg.h and re-compile. At present,
\c H5C__HASH_TABLE_LEN must be a power of two.
The \c epoch_length is the number of cache accesses between runs of the adaptive
cache size control algorithms. It is ignored if these algorithms are turned
off. It must lie in the range \Code{[H5C__MIN_AR_EPOCH_LENGTH,
H5C__MAX_AR_EPOCH_LENGTH]} -- currently [100, 1000000]. The above constants are
defined in \c H5Cprivate.h. 50000 is a reasonable value.
\subsection increment Increment Configuration
The \ref H5AC_cache_config_t.incr_mode "incr_mode" field specifies the cache
size increment algorithm used. Its value must be a member of the \ref
H5C_cache_incr_mode enum type -- currently either #H5C_incr__off or
#H5C_incr__threshold (note the double underscores after \c "incr"). This type is
defined in H5Cpublic.h.
If \ref H5AC_cache_config_t.incr_mode "incr_mode" is set to #H5C_incr__off,
regular automatic cache size increases are disabled, and the \ref
H5AC_cache_config_t.lower_hr_threshold "lower_hr_threshold", \ref
H5AC_cache_config_t.increment "increment", \ref
H5AC_cache_config_t.apply_max_increment "apply_max_increment", and \ref
H5AC_cache_config_t.max_increment "max_increment", fields are ignored.
The \ref H5AC_cache_config_t.flash_incr_mode "flash_incr_mode" field specifies
the flash cache size increment algorithm used. Its value must be a member of the
\ref H5C_cache_flash_incr_mode enum type -- currently either
#H5C_flash_incr__off or #H5C_flash_incr__add_space (note the double underscores
after \c "incr"). This type is defined in H5Cpublic.h.
If \ref H5AC_cache_config_t.flash_incr_mode "flash_incr_mode" is set to
#H5C_flash_incr__off, flash cache size increases are disabled, and the \ref
H5AC_cache_config_t.flash_multiple "flash_multiple", and \ref
H5AC_cache_config_t.flash_threshold "flash_threshold", fields are ignored.
\subsubsection hrtcsic Hit Rate Threshold Cache Size Increase Configuration
If \ref H5AC_cache_config_t.incr_mode "incr_mode" is #H5C_incr__threshold, the
cache size is increased via the hit rate threshold algorithm. The remaining
fields in the section are then used as follows:
\ref H5AC_cache_config_t.lower_hr_threshold "lower_hr_threshold" is the
threshold below which the hit rate must fall to trigger an increase. The value
must lie in the range \Code{[0.0 - 1.0]}. In my tests, a relatively high value
seems to work best -- 0.9 for example.
\ref H5AC_cache_config_t.increment "increment" is the factor by which the old
maximum cache size is multiplied to obtain an initial new maximum cache size
when an increment is needed. The actual change in size may be smaller as
required by \ref H5AC_cache_config_t.max_size "max_size" (above) and \c
max_increment (discussed below). increment must be greater than or equal to
1.0. If you set it to 1.0, you will effectively turn off the increment code. 2.0
is a reasonable value.
\ref H5AC_cache_config_t.apply_max_increment "apply_max_increment" and \ref
H5AC_cache_config_t.max_increment "max_increment" allow the user to specify a
maximum increment. If \ref H5AC_cache_config_t.apply_max_increment
"apply_max_increment" is \c TRUE, the cache size will never be increased by more
than the number of bytes specified in \ref H5AC_cache_config_t.max_increment
"max_increment" in any single increase.
\subsubsection fcsic Flash Cache Size Increase Configuration
If \ref H5AC_cache_config_t.flash_incr_mode "flash_incr_mode" is set to
#H5C_flash_incr__add_space, flash cache size increases are enabled. The size of
the cache will be increased under the following circumstances:
Let \c t be the current maximum cache size times the value of the \ref
H5AC_cache_config_t.flash_threshold "flash_threshold" field.
Let \c x be either the size of the newly inserted entry, the size of the newly
loaded entry, or the number of bytes added to the size of the entry under
consideration for triggering a flash cache size increase.
If \Code{t < x}, the basic condition for a flash cache size increase is met, and
we proceed as follows:
Let \c space_needed equal \c x less the amount of free space in the cache.
Further, let \ref H5AC_cache_config_t.increment "increment" equal \c
space_needed times the value of the \ref H5AC_cache_config_t.flash_multiple
"flash_multiple" field. If \ref H5AC_cache_config_t.increment "increment" plus
the current cache size is greater than \ref H5AC_cache_config_t.max_size
"max_size" (discussed above), reduce \ref H5AC_cache_config_t.increment
"increment" so that \ref H5AC_cache_config_t.increment "increment" plus the
current cache size is equal to \ref H5AC_cache_config_t.max_size "max_size".
If the increment is greater than zero, increase the current cache size by \ref
H5AC_cache_config_t.increment "increment". To avoid confusing the other cache
size increment or decrement algorithms, start a new epoch. Note, however, that
we do not cycle the epoch markers if some variant of the age out algorithm is in
use.
The use of the \ref H5AC_cache_config_t.flash_threshold "flash_threshold" field
is discussed above. It must be a floating-point value in the range of
\Code{[0.1, 1.0]}. 0.25 is a reasonable value.
The use of the \ref H5AC_cache_config_t.flash_multiple "flash_multiple" field is
also discussed above. It must be a floating-point value in the range of
\Code{[0.1, 10.0]}. 1.4 is a reasonable value.
\subsection decrement Decrement Configuration
The \ref H5AC_cache_config_t.decr_mode "decr_mode" field specifies the cache
size decrement algorithm used. Its value must be a member of the \ref
H5C_cache_decr_mode enum type -- currently either #H5C_decr__off,
#H5C_decr__threshold, #H5C_decr__age_out, or #H5C_decr__age_out_with_threshold
(note the double underscores after \c "decr"). This type is defined in
H5Cpublic.h.
If \ref H5AC_cache_config_t.decr_mode "decr_mode" is set to #H5C_decr__off,
automatic cache size decreases are disabled, and the remaining fields in the
cache size decrease control section are ignored.
\subsubsection hrtcsdc Hit Rate Threshold Cache Size Decrease Configuration
If \ref H5AC_cache_config_t.decr_mode "decr_mode" is #H5C_decr__threshold, the
cache size is decreased by the threshold algorithm, and the remaining fields of
the decrement section are used as follows:
\ref H5AC_cache_config_t.upper_hr_threshold "upper_hr_threshold" is the
threshold above which the hit rate must rise to trigger cache size reduction. It
must be in the range \Code{[0.0, 1.0]}. In my synthetic tests, very high values
like .9995 or .99995 seemed to work best.
\ref H5AC_cache_config_t.decrement "decrement" is the factor by which the
current maximum cache size is multiplied to obtain a tentative new maximum cache
size. It must lie in the range \Code{[0.0, 1.0]}. Relatively large values like
.9 seem to work best in my synthetic tests. Note that the actual size reduction
may be smaller as required by \ref H5AC_cache_config_t.min_size "min_size" and
\ref H5AC_cache_config_t.max_decrement "max_decrement" (discussed below). \ref
H5AC_cache_config_t.apply_max_decrement "apply_max_decrement" and \ref
H5AC_cache_config_t.max_decrement "max_decrement" allow the user to specify a
maximum decrement. If \ref H5AC_cache_config_t.apply_max_decrement
"apply_max_decrement" is \c TRUE, the cache size will never be reduced by more
than \ref H5AC_cache_config_t.max_decrement "max_decrement" bytes in any single
reduction.
With the hit rate threshold cache size decrement algorithm, the remaining fields
in the section are ignored.
\subsubsection acsr Ageout Cache Size Reduction
If \ref H5AC_cache_config_t.decr_mode "decr_mode" is #H5C_decr__age_out the
cache size is decreased by the ageout algorithm, and the remaining fields of the
decrement section are used as follows:
\ref H5AC_cache_config_t.epochs_before_eviction "epochs_before_eviction" is the
number of epochs an entry must reside unaccessed in the cache before it is
evicted. This value must lie in the range \Code{[1, H5C__MAX_EPOCH_MARKERS]}. \c
H5C__MAX_EPOCH_MARKERS is defined in H5Cprivate.h, and is currently set to 10.
\ref H5AC_cache_config_t.apply_max_decrement "apply_max_decrement" and \ref
H5AC_cache_config_t.max_decrement "max_decrement" are used as in section
2.4.3.1.
\ref H5AC_cache_config_t.apply_empty_reserve "apply_emty_reserve" and \ref
H5AC_cache_config_t.empty_reserve "empty_reserve" allow the user to specify a
minimum empty reserve as discussed in section 2.3.2.2. An empty reserve of 0.05
or 0.1 seems to work well.
The \ref H5AC_cache_config_t.decrement "decrement" and \ref
H5AC_cache_config_t.upper_hr_threshold "upper_hr_threshold" fields are ignored
in this case.
\subsubsection awhrtcsr Ageout With Hit Rate Threshold Cache Size Reduction
If \ref H5AC_cache_config_t.decr_mode "decr_mode" is
#H5C_decr__age_out_with_threshold, the cache size is decreased by the ageout
with hit rate threshold algorithm, and the fields of decrement section are used
as per the Ageout algorithm (see 5.3.2) with the exception of \ref
H5AC_cache_config_t.upper_hr_threshold "upper_hr_threshold".
Here, \ref H5AC_cache_config_t.upper_hr_threshold "upper_hr_threshold" is the
threshold above which the hit rate must rise to trigger cache size reduction. It
must be in the range \Code{[0.0, 1.0]}. In my synthetic tests, high values like
.999 seemed to work well.
\subsection parallel Parallel Configuration
This section is a catch-all for parallel specific configuration data. At
present, it has only one field --
\ref H5AC_cache_config_t.dirty_bytes_threshold "dirty_bytes_threshold".
In PHDF5, all operations that modify metadata must be executed collectively. We
used to think that this was enough to ensure consistency across the metadata
caches, but since we allow processes to read metadata individually, the order of
dirty entries in the LRU list can vary across processes. This, in turn, can
change the order in which dirty metadata cache entries reach the bottom of the
LRU and are flushed to disk -- opening the door to messages from the past and
messages from the future bugs.
To prevent this, only the metadata cache on process 0 of the file communicator
is allowed to write to file, and then only after entering a sync point with the
other caches. After it writes entries to file, it sends the base addresses of
the now clean entries to the other caches, so they can mark these entries clean
as well, and then leaves the sync point. The other caches mark the specified
entries as clean before they leave the synch point as well. (Observe, that since
all caches see the same stream of dirty metadata, they will all have the same
set of dirty entries upon sync point entry and exit.)
The different caches know when to synchronize by counting the number of bytes of
dirty metadata created by the collective operations modifying metadata. Whenever
this count exceeds the value specified in the \ref
H5AC_cache_config_t.dirty_bytes_threshold "dirty_bytes_threshold", they all
enter the sync point, and process 0 flushes down to its minimum clean size and
sends the list of newly cleaned entries to the other caches.
Needless to say, the value of the \ref H5AC_cache_config_t.dirty_bytes_threshold
"dirty_bytes_threshold" field must be consistent across all the caches operating
on a given file.
All dirty metadata can also by flushed under programmatic control via the
H5Fflush() call. This call must be collective and will reset the dirty data
counts on each metadata cache.
Absent calls to H5Fflush(), dirty metadata will only be flushed when the \ref
H5AC_cache_config_t.dirty_bytes_threshold "dirty_bytes_threshold" is exceeded,
and then only down to the H5AC_cache_config_t.min_clean_fraction
"min_clean_fraction". Thus, if a program does all its metadata modifications in
one phase, and then doesn't modify metadata thereafter, a residue of dirty
metadata will be frozen in the metadata caches for the remainder of the
computation -- effectively reducing the sizes of the caches.
In the default configuration, the caches will eventually resize themselves to
maintain an acceptable hit rate. However, this will take time, and it will
increase the application's footprint in memory.
If your application behaves in this manner, you can avoid this by a collective
call to H5Fflush() immediately after the metadata modification phase.
\subsection interactions Interactions
Evictions may not be disabled unless the automatic cache resize code is disabled
as well (by setting \ref H5AC_cache_config_t.decr_mode "decr_mode" to
#H5C_decr__off, \c flash_decr_mode to #H5C_flash_incr__add_space, and \ref
H5AC_cache_config_t.incr_mode "incr_mode" to #H5C_incr__off) -- thus placing the
cache size under the direct control of the user program.
There is no logical necessity for this restriction. It is imposed because it
simplifies testing greatly and because I can't see any reason why one would want
to disable evictions while the automatic cache size adjustment code was
enabled. This restriction can be relaxed if anyone can come up with a good
reason to do so.
At present, there are two interactions between the increment and decrement
sections of the configuration.
If \ref H5AC_cache_config_t.incr_mode "incr_mode" is #H5C_incr__threshold, and
\ref H5AC_cache_config_t.decr_mode "decr_mode" is either #H5C_decr__threshold or
#H5C_decr__age_out_with_threshold, then \ref
H5AC_cache_config_t.lower_hr_threshold "lower_hr_threshold" must be strictly
less than \ref H5AC_cache_config_t.upper_hr_threshold "upper_hr_threshold".
Also, if the flash cache size increment code is enabled and is triggered, it
will restart the current epoch without calling any other cache size increment or
decrement code.
In both the serial and parallel cases, there is the potential for an interaction
between the \ref H5AC_cache_config_t.min_clean_fraction "min_clean_fraction" and
the cache size increment code that can severely degrade
performance. Specifically, if the \ref H5AC_cache_config_t.min_clean_fraction
"min_clean_fraction" is large enough, it is possible that keeping the specified
fraction of the cache clean may generate enough flushes to seriously degrade
performance even though the hit rate is excellent.
In the serial case, this is easily dealt with by selecting a very small \ref
H5AC_cache_config_t.min_clean_fraction "min_clean_fraction" -- 0.01 for example
-- as this still avoids the "metadata blizzard" phenomenon that appears when the
cache fills with dirty metadata and must then flush all of it before evicting an
entry to make space for a new entry.
The problem is more difficult in the parallel case, as the \ref
H5AC_cache_config_t.min_clean_fraction "min_clean_fraction" is used to ensure
that the cache contains clean entries that can be evicted to make space for new
entries when metadata writes are forbidden -- i.e. between sync points.
This issue was discovered shortly before release 1.8.3 and an automated solution
has not been implemented. Should it become an issue for an application, try
manually setting the cache size to ~1.5 times the maximum working set size for
the application, and leave \ref H5AC_cache_config_t.min_clean_fraction
"min_clean_fraction" set to 0.3.
You can approximate the working set size of your application via repeated calls
to H5Fget_mdc_size() and H5Fget_mdc_hit_rate() while running your program with
the cache resize code enabled. The maximum value returned by H5Fget_mdc_size()
should be a reasonable approximation -- particularly if the associated hit rate
is good. In the parallel case, there is also an interaction between \c
min_clean_fraction and \ref H5AC_cache_config_t.dirty_bytes_threshold
"dirty_bytes_threshold". Absent calls to H5Fflush() (discussed above), the upper
bound on the amount of dirty data in the metadata caches will oscillate between
(1 - \ref H5AC_cache_config_t.min_clean_fraction "min_clean_fraction") times
current maximum cache size, and that value plus the \ref
H5AC_cache_config_t.dirty_bytes_threshold "dirty_bytes_threshold". Needless to
say, it will be best if the \ref H5AC_cache_config_t.min_size "min_size", \ref
H5AC_cache_config_t.min_clean_fraction "min_clean_fraction", and the \ref
H5AC_cache_config_t.dirty_bytes_threshold "dirty_bytes_threshold"
are chosen so that the cache can't fill with dirty data.
\subsection defaults Default Metadata Cache Configuration
Starting with release 1.8.3, HDF5 provides different default metadata cache
configurations depending on whether the library is compiled for serial or
parallel.
The default configuration for the serial case is as follows:
\code{.c}
{
/* int version = */ H5C__CURR_AUTO_SIZE_CTL_VER,
/* hbool_t rpt_fcn_enabled = */ FALSE,
/* hbool_t open_trace_file = */ FALSE,
/* hbool_t close_trace_file = */ FALSE,
/* char trace_file_name[] = */ "",
/* hbool_t evictions_enabled = */ TRUE,
/* hbool_t set_initial_size = */ TRUE,
/* size_t initial_size = */ ( 2 * 1024 * 1024),
/* double min_clean_fraction = */ 0.01,
/* size_t max_size = */ (32 * 1024 * 1024),
/* size_t min_size = */ ( 1 * 1024 * 1024),
/* long int epoch_length = */ 50000,
/* enum H5C_cache_incr_mode incr_mode = */ H5C_incr__threshold,
/* double lower_hr_threshold = */ 0.9,
/* double increment = */ 2.0,
/* hbool_t apply_max_increment = */ TRUE,
/* size_t max_increment = */ (4 * 1024 * 1024),
/* enum H5C_cache_flash_incr_mode */
/* flash_incr_mode = */ H5C_flash_incr__add_space,
/* double flash_multiple = */ 1.4,
/* double flash_threshold = */ 0.25,
/* enum H5C_cache_decr_mode decr_mode = */ H5C_decr__age_out_with_threshold,
/* double upper_hr_threshold = */ 0.999,
/* double decrement = */ 0.9,
/* hbool_t apply_max_decrement = */ TRUE,
/* size_t max_decrement = */ (1 * 1024 * 1024),
/* int epochs_before_eviction = */ 3,
/* hbool_t apply_empty_reserve = */ TRUE,
/* double empty_reserve = */ 0.1,
/* int dirty_bytes_threshold = */ (256 * 1024)
}
\endcode
The default configuration for the parallel case is as follows:
\code{.c}
{
/* int version = */ H5C__CURR_AUTO_SIZE_CTL_VER,
/* hbool_t rpt_fcn_enabled = */ FALSE,
/* hbool_t open_trace_file = */ FALSE,
/* hbool_t close_trace_file = */ FALSE,
/* char trace_file_name[] = */ "",
/* hbool_t evictions_enabled = */ TRUE,
/* hbool_t set_initial_size = */ TRUE,
/* size_t initial_size = */ ( 2 * 1024 * 1024),
/* double min_clean_fraction = */ 0.3,
/* size_t max_size = */ (32 * 1024 * 1024),
/* size_t min_size = */ ( 1 * 1024 * 1024),
/* long int epoch_length = */ 50000,
/* enum H5C_cache_incr_mode incr_mode = */ H5C_incr__threshold,
/* double lower_hr_threshold = */ 0.9,
/* double increment = */ 2.0,
/* hbool_t apply_max_increment = */ TRUE,
/* size_t max_increment = */ (4 * 1024 * 1024),
/* enum H5C_cache_flash_incr_mode */
/* flash_incr_mode = */ H5C_flash_incr__add_space,
/* double flash_multiple = */ 1.0,
/* double flash_threshold = */ 0.25,
/* enum H5C_cache_decr_mode decr_mode = */ H5C_decr__age_out_with_threshold,
/* double upper_hr_threshold = */ 0.999,
/* double decrement = */ 0.9,
/* hbool_t apply_max_decrement = */ TRUE,
/* size_t max_decrement = */ (1 * 1024 * 1024),
/* int epochs_before_eviction = */ 3,
/* hbool_t apply_empty_reserve = */ TRUE,
/* double empty_reserve = */ 0.1,
/* int dirty_bytes_threshold = */ (256 * 1024)
}
\endcode
The default serial configuration should be adequate for most serial HDF5 users.
The same may not be true for the default parallel configuration due to the
interaction between the \ref H5AC_cache_config_t.min_clean_fraction "min_clean_fraction" and the cache size increase code. See
the Interactions section for further details.
Should you need to change the default configuration, it can be found in
H5ACprivate.h. Look for the definition of H5AC__DEFAULT_RESIZE_CONFIG.
\section controlling Controlling the New Metadata Cache Size From Your Program
You have already seen how \ref H5AC_cache_config_t has facilities that allow you
to control the metadata cache size directly. Use H5Fget_mdc_config() and
H5Fset_mdc_config() to get and set the metadata cache configuration on an open
file. Use H5Pget_mdc_config() and H5Pset_mdc_config() to get and set the initial
metadata cache configuration in a file access property list. Recall that this
list contains configuration data used when opening a file.
Use H5Fget_mdc_hit_rate() to get the average hit rate since the last time the
hit rate stats were reset. This happens automatically at the beginning of each
epoch if the adaptive cache resize code is enabled. You can also do it manually
with H5Freset_mdc_hit_rate_stats(). Be careful about doing this if the adaptive
cache resize code is enabled, as you may confuse it.
Use H5Fget_mdc_size() to get metadata cache size data on an open file.
Finally, note that cache size and cache footprint are two different things -- in
my tests, the cache footprint (as inferred from the UNIX \c top command) is
typically about three times the maximum cache size. I haven't tracked it down
yet, but I would guess that most of this is due to the very small typical cache
entry size combined with the rather large size of the cache entry header
structure. This should be investigated further, but there are other matters of
higher priority.
\section news New Metadata Cache Debugging Facilities
The new metadata cache has a variety of debugging facilities that may be of
use. I doubt that any other than the report function and the trace file will
ever be accessible via the API, but they are relatively easy to turn on in the
source code.
Note that none of this should be viewed as supported -- it is described here on
the off chance that you want to use it, but you are on your own if you do. Also,
there are no promises as to consistency between versions.
As mentioned above, you can use the \ref H5AC_cache_config_t.rpt_fcn_enabled "rpt_fcn_enabled" field of the
configuration structure to enable the default reporting function
(H5C_def_auto_resize_rpt_fcn() in H5C.c). If this function doesn't work for you,
you will have to write your own. In particular, remember that it uses \c stdout,
so it will probably be unhappy under Windows.
Again, remember that this facility is not supported. Further, it is likely to
change every time I do any serious work on the cache.
There is also an extensive statistics collection code. Use
H5C_COLLECT_CACHE_STATS and H5C_COLLECT_CACHE_ENTRY_STATS in H5Cprivate.h to
turn this on. If you also turn on H5AC_DUMP_STATS_ON_CLOSE in H5ACprivate.h,
stats will be dumped when you close a file. Alternatively you can call
H5C_stats() and H5C_stats__reset() within the library to dump and reset
stats. Both of these functions are defined in H5C.c.
Finally, the cache also contains an extensive sanity checking code. Much of this
is turned on when you compile in debug mode, but to enable the full suite, turn
on H5C_DO_SANITY_CHECKS in H5Cprivate.h.
\section trouble Trouble Shooting
Absent major bugs in the cache, the only troubleshooting you should have to do
is diagnosing and fixing problems with your cache configuration.
Assuming it runs on your platform (I've only used it under Linux), the reporting
function is probably the most convenient diagnosis tool. However, since it is
unsupported code, I will not discuss it further beyond directing you to the
source (H5C_def_auto_resize_rpt_fcn() in H5C.c).
Absent the reporting function, regular calls to H5Fget_mdc_hit_rate() should
give you a good idea of the hit rate over time. Remember that the hit rate stats
are reset at the end of each epoch (when adaptive cache resizing is enabled), so
you should expect some jitter.
Similar calls to H5Fget_mdc_size() should allow you to monitor cache size and
the fraction of the current maximum cache size that is actually in use.
If the hit rate is consistently low, and the cache it at its current maximum
size, increasing the maximum size is an obvious fix.
If you see hit rate and cache size oscillations, try disabling adaptive cache
resizing and setting a fixed cache size a bit greater than the high end of the
cache size oscillations you observed.
If the hit rate oscillations don't go away, you are probably looking at a
feature of your application that can't be helped without major changes to the
cache. Please send along a description of the situation.
If the oscillations do go away, you may be able to come up with a configuration
that deals with the situation. If that fails, control the cache size manually,
and write to me, so I can try to develop an adaptive resize algorithm that works
in your case.
Needless to say, you should give the cache a few epochs to adapt to
circumstances. If that is too slow for you, try manual cache size control.
If you find it necessary to disable evictions, you may find it useful to enable
the internal statistics collection code mentioned above in the section on
debugging facilities.
Amongst many other things, the stats code will report the maximum cache size,
and the average successful and unsuccessful search depths in the hash table. If
these latter figures are significantly above 1, you should increase the size of
the hash table.
*/
doxygen/dox/OtherSpecs.dox
+11
View File
@@ -0,0 +1,11 @@
/** \page IMG HDF5 Image and Palette Specification Version 1.2
\htmlinclude ImageSpec.html
*/
/** \page TBL HDF5 Table Specification Version 1.0
\htmlinclude TableSpec.html
*/
doxygen/dox/Overview.dox
+32
View File
@@ -0,0 +1,32 @@
/** \mainpage notitle
This is the documentation set for HDF5. You can
<a href="hdf5-doc.tgz">download</a> it as a tgz archive for offline reading.
This is the documention set for HDF5 in terms of specifications and software
developed and maintained by <a href="https://www.hdfgroup.org/">The HDF
Group</a>. It is impractical to document the entire HDF5 ecosystem in one place,
and you should also consult the documentation sets of the many outstanding
community projects.
For a first contact with HDF5, the best place is to have a look at the \link
GettingStarted getting started \endlink page that shows you how to write and
compile your first program with HDF5.
The \b main \b documentation is organized by documentation flavor. Most
technical documentation consists to varying degrees of information related to
<em>tasks</em>, <em>concepts</em>, or <em>reference</em> material. As its title
suggests, the \link RM Reference Manual \endlink is 100% reference material,
while the \link Cookbook \endlink is focused on tasks. The different guide-type
documents cover a mix of tasks, concepts, and reference, to help a certain
<em>audience</em> succeed.
Finally, do not miss the search engine (top right-hand corner)! If you are
looking for a specific function, it'll take you there directly. If unsure, it'll
give you an idea of what's on offer and a few promising leads.
\par ToDo List
There is plenty of <a href="./todo.html">unfinished business</a>.
*/
doxygen/dox/ReferenceManual.dox
+43
View File
@@ -0,0 +1,43 @@
/** \page RM Reference Manual
The functions provided by the HDF5 C-API are grouped into the following
\Emph{modules}:
\li \ref H5A "Attributes" — Management of HDF5 attributes (\ref H5A)
\li \ref H5D "Datasets" — Management of HDF5 datasets (\ref H5D)
\li \ref H5S "Dataspaces" — Management of HDF5 dataspaces which describe the shape of datasets and attributes (\ref H5S)
\li \ref H5T "Datatypes" — Management of datatypes which describe elements of datasets and attributes (\ref H5T)
\li \ref H5E "Error Handling" — Functions for handling HDF5 errors (\ref H5E)
\li \ref H5ES "Event Sets" — Functions for handling HDF5 event sets (\ref H5ES)
\li \ref H5F "Files" — Management of HDF5 files (\ref H5F)
\li \ref H5Z "Filters" — Configuration of filters that process data during I/O operation (\ref H5Z)
\li \ref H5G "Groups" — Management of groups in HDF5 files (\ref H5G)
\li \ref H5I "Identifiers" — Management of object identifiers and object names (\ref H5I)
\li \ref H5 "Library" — General purpose library functions (\ref H5)
\li \ref H5L "Links" — Management of links in HDF5 groups (\ref H5L)
\li \ref H5M "Maps" — Management of HDF5 maps (\ref H5M)
\li \ref H5O "Objects" — Management of objects in HDF5 files (\ref H5O)
\li \ref H5PL "Plugins" — Programmatic control over dynamically loaded plugins (\ref H5PL)
\li \ref H5P "Property Lists" — Management of property lists to control HDF5 library behavior (\ref H5P)
\li \ref H5R "References" — Management of references to specific objects and data regions in an HDF5 file (\ref H5R)
\li \ref H5VL "Virtual Object Layer" — Management of the Virtual Object Layer (\ref H5VL)
\par Asynchronous Functions
A subset of functions has \ref ASYNC "asynchronous variants".
\par API Versioning
See \ref api-compat-macros
\par Deprecated Functions and Types
A list of deprecated functions and types can be found
<a href="./deprecated.html">here</a>.
\par Etiquette
Here are a few simple rules to follow:
\li \Bold{Handle discipline:} If you acquire a handle (by creation or copy), \Emph{you own it!} (..., i.e., you have to close it.)
\li \Bold{Dynamic memory allocation:} ...
\li \Bold{Use of locations:} Identifier + name combo
\cpp_c_api_note
*/
doxygen/dox/Specifications.dox
+22
View File
@@ -0,0 +1,22 @@
/** \page SPEC Specifications
\section DDL
\li \ref DDLBNF110 "DDL in BNF through HDF5 1.10"
\li \ref DDLBNF112 "DDL in BNF for HDF5 1.12 and above"
\section File Format
\li \ref FMT1 "HDF5 File Format Specification Version 1.0"
\li \ref FMT11 "HDF5 File Format Specification Version 1.1"
\li \ref FMT2 "HDF5 File Format Specification Version 2.0"
\li \ref FMT3 "HDF5 File Format Specification Version 3.0"
\section Other
\li \ref IMG "HDF5 Image and Palette Specification Version 1.2"
\li \ref TBL "HDF5 Table Specification Version 1.0"
\li <a href="https://support.hdfgroup.org/HDF5/doc/HL/H5DS_Spec.pdf">
HDF5 Dimension Scale Specification</a>
*/
doxygen/dox/TechnicalNotes.dox
+20
View File
@@ -0,0 +1,20 @@
/** \page TN Technical Notes
\li \link api-compat-macros API Compatibility Macros \endlink
\li \ref TNMDC "Metadata Caching in HDF5"
\li \ref MT "Thread Safe library"
\li \ref VFL "Virtual File Layer"
*/
/** \page MT HDF5 Thread Safe library
\htmlinclude ThreadSafeLibrary.html
*/
/** \page VFL HDF5 Virtual File Layer
\htmlinclude VFL.html
*/
doxygen/dox/api-compat-macros.dox
-1
View File
@@ -1,5 +1,4 @@
/** \page api-compat-macros API Compatibility Macros
\tableofcontents
\section audience Audience
The target audience for this document has existing applications that use the
doxygen/dox/mainpage.dox
-44
View File
@@ -1,44 +0,0 @@
/*! \mainpage HDF5 C-API Reference
*
* The HDF5 C-API provides applications with fine-grained control over all
* aspects HDF5 functionality. This functionality is grouped into the following
* \Emph{modules}:
* \li \ref H5A "Attributes" — Management of HDF5 attributes (\ref H5A)
* \li \ref H5D "Datasets" — Management of HDF5 datasets (\ref H5D)
* \li \ref H5S "Dataspaces" — Management of HDF5 dataspaces which describe the shape of datasets and attributes (\ref H5S)
* \li \ref H5T "Datatypes" — Management of datatypes which describe elements of datasets and attributes (\ref H5T)
* \li \ref H5E "Error Handling" — Functions for handling errors that occur within HDF5 (\ref H5E)
* \li \ref H5F "Files" — Management of HDF5 files (\ref H5F)
* \li \ref H5Z "Filters" — Configuration of filters that process data during I/O operation (\ref H5Z)
* \li \ref H5G "Groups" — Management of groups in HDF5 files (\ref H5G)
* \li \ref H5I "Identifiers" — Management of object identifiers and object names (\ref H5I)
* \li \ref H5 "Library" — General purpose library functions (\ref H5)
* \li \ref H5L "Links" — Management of links in HDF5 groups (\ref H5L)
* \li \ref H5O "Objects" — Management of objects in HDF5 files (\ref H5O)
* \li \ref H5PL "Plugins" — Programmatic control over dynamically loaded plugins (\ref H5PL)
* \li \ref H5P "Property Lists" — Management of property lists to control HDF5 library behavior (\ref H5P)
* \li \ref H5R "References" — Management of references to specific objects and data regions in an HDF5 file (\ref H5R)
* \li \ref H5VL "Virtual Object Layer" — Management of the Virtual Object Layer (\ref H5VL)
*
* Here are a few simple rules to follow:
*
* \li \Bold{Handle discipline:} If you acquire a handle (by creation or coopy), \Emph{you own it!} (..., i.e., you have to close it.)
* \li \Bold{Dynamic memory allocation:} ...
* \li \Bold{Use of locations:} Identifier + name combo
*
* \attention \Bold{C++ Developers using HDF5 C-API functions beware:}\n
* If a C routine that takes a function pointer as an argument is called from
* within C++ code, the C routine should be returned from normally.
* Examples of this kind of routine include callbacks such as H5Pset_elink_cb()
* and H5Pset_type_conv_cb() and functions such as H5Tconvert() and H5Ewalk2().\n
* Exiting the routine in its normal fashion allows the HDF5 C library to clean
* up its work properly. In other words, if the C++ application jumps out of
* the routine back to the C++ \c catch statement, the library is not given the
* opportunity to close any temporary data structures that were set up when the
* routine was called. The C++ application should save some state as the
* routine is started so that any problem that occurs might be diagnosed.
*
* \todo Fix the search form for server deployments.
* \todo Make it mobile-friendly
*
*/
doxygen/dox/maybe_metadata_reads.dox
+82
View File
@@ -0,0 +1,82 @@
/**
* \page maybe_metadata_reads Functions with No Access Property List Parameter that May Generate Metadata Reads
*
* \ingroup GACPL
*
* Currently there are several operations in HDF5 that can issue metadata reads
* from the metadata cache, but that take no property list. It is therefore not
* possible to set a collective requirement individually for those operations. The
* only solution with the HDF5 1.10.0 release is to set the collective requirement
* globally on H5Fopen() or H5Fcreate() for all metadata operations to be
* collective.
*
* The following is a list of those functions in the HDF5 library. This list is
* integral to the discussion in the H5Pset_all_coll_metadata_ops() entry:
*
* <pre>
*
* H5Awrite()
* H5Aread()
* H5Arename()
* H5Aiterate2()
* H5Adelete()
* H5Aexists()
*
* H5Dget_space_status()
* H5Dget_storage_size()
* H5Dset_extent()
* H5Ddebug()
* H5Dclose()
* H5Dget_create_plist()
* H5Dget_space() (when dataset is a virtual dataset)
*
* H5Gget_create_plist()
* H5Gget_info()
* H5Gclose()
*
* H5Literate()
* H5Lvisit()
*
* H5Rcreate()
* H5Rdereference2() (when reference is an object reference)
* H5Rget_region()
* H5Rget_obj_type2()
* H5Rget_name()
*
* H5Ocopy()
* H5Oopen_by_addr()
* H5Oincr_refcount()
* H5Odecr_refcount()
* H5Oget_info()
* H5Oset_comment()
* H5Ovisit()
*
* H5Fis_hdf5()
* H5Fflush()
* H5Fclose()
* H5Fget_file_image()
* H5Freopen()
* H5Fget_freespace()
* H5Fget_info2()
* H5Fget_free_sections()
* H5Fmount()
* H5Funmount()
*
* H5Iget_name()
*
* H5Tget_create_plist()
* H5Tclose()
*
* H5Zunregister()
* </pre>
*
* In addition, \b most deprecated functions fall into this category.
*
* The HDF Group may address the above limitation in a future major release, but
* no decision has been made at this time. Such a change might, for example,
* include adding new versions of some or all the above functions with an extra
* property list parameter to allow an individual setting for the collective
* calling requirement.
*
* \sa_metadata_ops
*/
doxygen/examples/FF-IH_FileGroup.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.3 KiB

doxygen/examples/FF-IH_FileObject.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.1 KiB

doxygen/examples/FileFormatSpecChunkDiagram.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

doxygen/examples/H5.format.1.0.html
+4050
View File
@@ -0,0 +1,4050 @@
<html>
<head>
<title>
HDF5 File Format Specification
</title>
</head>
<body bgcolor="#FFFFFF">
<center>
<table border=0 width=90%>
<tr>
<td valign=top>
<ol type=I>
<li><a href="#Intro">Introduction</a>
<li><a href="#BootBlock">Disk Format Level 0 - File Signature and Super Block</a>
<li><a href="#Group">Disk Format Level 1 - File Infrastructure</a>
<font size=-2>
<ol type=A>
<li><a href="#Btrees">Disk Format Level 1A - B-link Trees and B-tree Nodes</a>
<li><a href="#SymbolTable">Disk Format Level 1B - Group</a>
<li><a href="#SymbolTableEntry">Disk Format Level 1C - Group Entry</a>
<li><a href="#LocalHeap">Disk Format Level 1D - Local Heaps</a>
<li><a href="#GlobalHeap">Disk Format Level 1E - Global Heap</a>
<li><a href="#FreeSpaceIndex">Disk Format Level 1F - Free-space Index</a>
</ol>
</font>
<li><a href="#DataObject">Disk Format Level 2 - Data Objects</a>
<font size=-2>
<ol type=A>
<li><a href="#ObjectHeader">Disk Format Level 2a - Data Object Headers</a>
<ol type=1>
<li><a href="#NILMessage">Name: NIL</a> <!-- 0x0000 -->
<li><a href="#SimpleDataSpace">Name: Simple Dataspace</a> <!-- 0x0001 -->
<!--
<li><a href="#DataSpaceMessage">Name: Complex Dataspace</a> --> <!-- 0x0002 -->
<li><a href="#DataTypeMessage">Name: Datatype</a> <!-- 0x0003 -->
<li><a href="#FillValueMessage">Name: Data Storage - Fill Value</a> <!-- 0x0004 -->
<li><a href="#ReservedMessage_0005">Name: Reserved - not assigned yet</a> <!-- 0x0005 -->
</ol>
</ol>
</font>
</ol>
</td><td>&nbsp;&nbsp;</td><td valign=top>
<ol type=I>
<li><a href="#DataObject">Disk Format Level 2 - Data Objects</a>
<font size=-2><i>(Continued)</i>
<ol type=A>
<li><a href="#ObjectHeader">Disk Format Level 2a - Data Object Headers</a><i>(Continued)</i>
<ol type=1>
<li><a href="#CompactDataStorageMessage">Name: Data Storage - Compact</a> <!-- 0x0006 -->
<li><a href="#ExternalFileListMessage">Name: Data Storage - External Data Files</a> <!-- 0x0007 -->
<li><a href="#LayoutMessage">Name: Data Storage - Layout</a> <!-- 0x0008 -->
<li><a href="#ReservedMessage_0009">Name: Reserved - not assigned yet</a> <!-- 0x0009 -->
<li><a href="#ReservedMessage_000A">Name: Reserved - not assigned yet</a> <!-- 0x000a -->
<li><a href="#FilterMessage">Name: Data Storage - Filter Pipeline</a> <!-- 0x000b -->
<li><a href="#AttributeMessage">Name: Attribute</a> <!-- 0x000c -->
<li><a href="#NameMessage">Name: Object Name</a> <!-- 0x000d -->
<li><a href="#ModifiedMessage">Name: Object Modification Date and Time</a> <!-- 0x000e -->
<li><a href="#SharedMessage">Name: Shared Object Message</a> <!-- 0x000f -->
<li><a href="#ContinuationMessage">Name: Object Header Continuation</a> <!-- 0x0010 -->
<li><a href="#SymbolTableMessage">Name: Group Message</a> <!-- 0x0011 -->
</ol>
<li><a href="#SharedObjectHeader">Disk Format: Level 2b - Shared Data Object Headers</a>
<li><a href="#DataStorage">Disk Format: Level 2c - Data Object Data Storage</a>
</ol>
</font>
</ol>
</td></tr>
</table>
</center>
<br><br>
<h2>Introduction</h2>
<table align=right width=100>
<tr><td>&nbsp;</td><td align=center>
<hr>
<img src="FF-IH_FileGroup.gif" alt="HDF5 Groups" hspace=15 vspace=15>
</td><td>&nbsp;</td></tr>
<tr><td>&nbsp;</td><td align=center>
<strong>Figure 1:</strong> Relationships among the HDF5 root group, other groups, and objects
<hr>
</td><td>&nbsp;</td></tr>
<tr><td>&nbsp;</td><td align=center>
<img src="FF-IH_FileObject.gif" alt="HDF5 Objects" hspace=15 vspace=15>
</td><td>&nbsp;</td></tr>
<tr><td>&nbsp;</td><td align=center>
<strong>Figure 2:</strong> HDF5 objects -- datasets, datatypes, or dataspaces
<hr>
</td><td>&nbsp;</td></tr>
</table>
<P>The format of an HDF5 file on disk encompasses several
key ideas of the HDF4 and AIO file formats as well as
addressing some shortcomings therein. The new format is
more self-describing than the HDF4 format and is more
uniformly applied to data objects in the file.
<P>An HDF5 file appears to the user as a directed graph.
The nodes of this graph are the higher-level HDF5 objects
that are exposed by the HDF5 APIs:
<ul>
<li>Groups
<li>Datasets
<li>Datatypes
<li>Dataspaces
</ul>
<P>At the lowest level, as information is actually written to the disk,
an HDF5 file is made up of the following objects:
<ul>
<li>A super block
<li>B-tree nodes (containing either symbol nodes or raw data chunks)
<li>Object headers
<li>Collections
<li>Local heaps
<li>Free space
</ul>
The HDF5 library uses these lower-level objects to represent the
higher-level objects that are then presented to the user or
to applications through the APIs.
For instance, a group is an object header that contains a message that
points to a local heap and to a B-tree which points to symbol nodes.
A dataset is an object header that contains messages that describe
datatype, space, layout, filters, external files, fill value, etc
with the layout message pointing to either a raw data chunk or to a
B-tree that points to raw data chunks.
<h3>This Document</h3>
<p>This document describes the lower-level data objects;
the higher-level objects and their properties are described
in the <a href="H5.user.html"><cite>HDF5 User's Guide</cite></a>.
<!--
<blockquote>
<pre>
Elena> NOTE: give reference to the detailed discussion of the B-trees
Elena> when needed. Right now we do not have specification (only general one)
Elena> for the Symbol Table B-trees and B-trees used to manage chunked datasets.
Elena> B-trees
Elena> General Discussion
Elena> Object related discussions
Elena> Symbol Tables
Elena> Global heap
Elena> "Free-space object"
</pre>
</blockquote>
-->
<P>Three levels of information comprise the file format.
Level 0 contains basic information for identifying and
defining information about the file. Level 1 information contains
the group information (stored as a B-tree) and is used as the
index for all the objects in the file. Level 2 is the rest
of the file and contains all of the data objects, with each object
partitioned into header information, also known as
<em>meta information</em>, and data.
<p>The sizes of various fields in the following layout tables are
determined by looking at the number of columns the field spans
in the table. There are three exceptions: (1) The size may be
overridden by specifying a size in parentheses, (2) the size of
addresses is determined by the <em>Size of Offsets</em> field
in the super block, and (3) the size of size fields is determined
by the <em>Size of Lengths</em> field in the super block.
<br><br>
<br><br>
<h2><a name="BootBlock">
Disk Format: Level 0 - File Signature and Super Block</a></h2>
<P>The super block may begin at certain predefined offsets within
the HDF5 file, allowing a block of unspecified content for
users to place additional information at the beginning (and
end) of the HDF5 file without limiting the HDF5 library's
ability to manage the objects within the file itself. This
feature was designed to accommodate wrapping an HDF5 file in
another file format or adding descriptive information to the
file without requiring the modification of the actual file's
information. The super block is located by searching for the
HDF5 file signature at byte offset 0, byte offset 512 and at
successive locations in the file, each a multiple of two of
the previous location, i.e. 0, 512, 1024, 2048, etc.
<P>The super block is composed of a file signature, followed by
super block and group version numbers, information
about the sizes of offset and length values used to describe
items within the file, the size of each group page,
and a group entry for the root object in the file.
<p>
<center>
<table border align=center cellpadding=4 width="80%">
<caption align=top>
<B>HDF5 Super Block Layout</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4><br>HDF5 File Signature (8 bytes)<br><br></td>
</tr>
<tr align=center>
<td>Version # of Super Block</td>
<td>Version # of Global Free-space Storage</td>
<td>Version # of Group</td>
<td>Reserved</td>
</tr>
<tr align=center>
<td>Version # of Shared Header Message Format</td>
<td>Size of Offsets</td>
<td>Size of Lengths</td>
<td>Reserved (zero)</td>
</tr>
<tr align=center>
<td colspan=2>Group Leaf Node K</td>
<td colspan=2>Group Internal Node K</td>
</tr>
<tr align=center>
<td colspan=4>File Consistency Flags</td>
</tr>
<tr align=center>
<td colspan=4>Base Address*</td>
</tr>
<tr align=center>
<td colspan=4>Address of Global Free-space Heap*</td>
</tr>
<tr align=center>
<td colspan=4>End of File Address*</td>
</tr>
<tr align=center>
<td colspan=4>Driver Information Block Address*</td>
</tr>
<tr align=center>
<td colspan=4>Root Group Address*</td>
</tr>
</table>
<table width="80%" border=0>
<tr><td>
<div align=right>
(Items marked with an asterisk (*) in the above table
<br>
are of the size specified in "Size of Offsets.")
</div>
</td></tr>
</table>
</center>
<p>
<center>
<table width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>File Signature</td>
<td>This field contains a constant value and can be used to
quickly identify a file as being an HDF5 file. The
constant value is designed to allow easy identification of
an HDF5 file and to allow certain types of data corruption
to be detected. The file signature of an HDF5 file always
contains the following values:
<br><br><center>
<table border align=center cellpadding=4 width="100%">
<tr align=center>
<td>decimal</td>
<td width="8%">137</td>
<td width="8%">72</td>
<td width="8%">68</td>
<td width="8%">70</td>
<td width="8%">13</td>
<td width="8%">10</td>
<td width="8%">26</td>
<td width="8%">10</td>
</tr>
<tr align=center>
<td>hexadecimal</td>
<td width="8%">89</td>
<td width="8%">48</td>
<td width="8%">44</td>
<td width="8%">46</td>
<td width="8%">0d</td>
<td width="8%">0a</td>
<td width="8%">1a</td>
<td width="8%">0a</td>
</tr>
<tr align=center>
<td>ASCII C Notation</td>
<td width="8%">\211</td>
<td width="8%">H</td>
<td width="8%">D</td>
<td width="8%">F</td>
<td width="8%">\r</td>
<td width="8%">\n</td>
<td width="8%">\032</td>
<td width="8%">\n</td>
</tr>
</table>
</center>
<br>
This signature both identifies the file as an HDF5 file
and provides for immediate detection of common
file-transfer problems. The first two bytes distinguish
HDF5 files on systems that expect the first two bytes to
identify the file type uniquely. The first byte is
chosen as a non-ASCII value to reduce the probability
that a text file may be misrecognized as an HDF5 file;
also, it catches bad file transfers that clear bit
7. Bytes two through four name the format. The CR-LF
sequence catches bad file transfers that alter newline
sequences. The control-Z character stops file display
under MS-DOS. The final line feed checks for the inverse
of the CR-LF translation problem. (This is a direct
descendent of the PNG file signature.)</td>
</tr>
<tr valign=top>
<td>Version Number of the Super Block</td>
<td>This value is used to determine the format of the
information in the super block. When the format of the
information in the super block is changed, the version number
is incremented to the next integer and can be used to
determine how the information in the super block is
formatted.</td>
</tr>
<tr valign=top>
<td>Version Number of the Global Free-space Heap</td>
<td>This value is used to determine the format of the
information in the Global Free-space Heap.</td>
</tr>
<tr valign=top>
<td>Version Number of the Group</td>
<td>This value is used to determine the format of the
information in the Group. When the format of
the information in the Group is changed, the
version number is incremented to the next integer and can be
used to determine how the information in the Group
is formatted.</td>
</tr>
<tr valign=top>
<td>Version Number of the Shared Header Message Format</td>
<td>This value is used to determine the format of the
information in a shared object header message, which is
stored in the global small-data heap. Since the format
of the shared header messages differs from the private
header messages, a version number is used to identify changes
in the format.</td>
</tr>
<tr valign=top>
<td>Size of Offsets</td>
<td>This value contains the number of bytes used to store
addresses in the file. The values for the addresses of
objects in the file are offsets relative to a base address,
usually the address of the super block signature. This
allows a wrapper to be added after the file is created
without invalidating the internal offset locations.</td>
</tr>
<tr valign=top>
<td>Size of Lengths</td>
<td>This value contains the number of bytes used to store
the size of an object.</td>
</tr>
<tr valign=top>
<td>Group Leaf Node K</td>
<td>Each leaf node of a group B-tree will have at
least this many entries but not more than twice this
many. If a group has a single leaf node then it
may have fewer entries.</td>
</tr>
<tr valign=top>
<td>Group Internal Node K</td>
<td>Each internal node of a group B-tree will have
at least K pointers to other nodes but not more than 2K
pointers. If the group has only one internal
node then it might have fewer than K pointers.</td>
</tr>
<tr valign=top>
<td>Bytes per B-tree Page</td>
<td>This value contains the number of bytes used for symbol
pairs per page of the B-trees used in the file. All
B-tree pages will have the same size per page.
<br>
For 32-bit file offsets, 340 objects is the maximum
per 4KB page; for 64-bit file offset, 254 objects will fit
per 4KB page. In general, the equation is:
<br>
<code>&nbsp;&nbsp;&nbsp;&lt;<i>number of objects</i>&gt; =
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
FLOOR((&lt;<i>page size</i>&gt; - &lt;<i>offset size</i>&gt;) /
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
(&lt;<i>Symbol size</i>&gt; + &lt;<i>offset size</i>&gt;))
- 1 </code></td>
</tr>
<tr valign=top>
<td>File Consistency Flags</td>
<td>This value contains flags to indicate information
about the consistency of the information contained
within the file. Currently, the following bit flags are
defined:
<ul>
<li>Bit 0 set indicates that the file is opened for
write-access.
<li>Bit 1 set indicates that the file has
been verified for consistency and is guaranteed to be
consistent with the format defined in this document.
<li>Bits 2-31 are reserved for future use.
</ul>
Bit 0 should be
set as the first action when a file is opened for write
access and should be cleared only as the final action
when closing a file. Bit 1 should be cleared during
normal access to a file and only set after the file's
consistency is guaranteed by the library or a
consistency utility.</td>
</tr>
<tr valign=top>
<td>Base Address</td>
<td>This is the absolute file address of the first byte of
the HDF5 data within the file. The library currently
constrains this value to be the absolute file address
of the super block itself when creating new files;
future versions of the library may provide greater
flexibility. Unless otherwise noted,
all other file addresses are relative to this base
address.</td>
</tr>
<tr valign=top>
<td>Address of Global Free-space Heap</td>
<td>Free-space management is not yet defined in the HDF5
file format and is not handled by the library.
Currently this field always contains the
undefined address <code>0xfff...ff</code>.
<!--
<td>This value contains the relative address of the B-tree
used to manage the blocks of data which are unused in the
file currently. The free-space heap is used to manage the
blocks of bytes at the file-level which become unused when
objects are moved within the file.</td>
-->
</tr>
<tr valign=top>
<td>End of File Address</td>
<td>This is the relative file address of the first byte past
the end of all HDF5 data. It is used to determine whether a
file has been accidently truncated and as an address where
file data allocation can occur if the free list is not
used.</td>
</tr>
<tr valign=top>
<td>Driver Information Block Address</td>
<td>This is the relative file address of the file driver
information block which contains driver-specific
information needed to reopen the file. If there is no
driver information block then this entry should be the
undefined address (all bits set).</td>
</tr>
<tr valign=top>
<td>Root Group Address</td>
<td>This is the address of the root group (described later
in this document), which serves as the entry point into
the group graph.</td>
</tr>
</table>
</center>
<p>The <em>file driver information block</em> is an optional region of the
file which contains information needed by the file driver in
order to reopen a file. The format of the file driver information
block is:
<p>
<center>
<table border align=center cellpadding=4 width="80%">
<caption align=top>
<B>Driver Information Block</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td>Version</td>
<td colspan=3>Reserved (zero)</td>
</tr>
<tr align=center>
<td colspan=4>Driver Information Size (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4><br>Driver Identification (8 bytes)<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br><br>Driver Information<br><br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version</td>
<td>The version number of the driver information block. The
file format documented here is version zero.</td>
</tr>
<tr valign=top>
<td>Driver Information Size</td>
<td>The size in bytes of the Driver Information part of this
structure.</td>
</tr>
<tr valign=top>
<td>Driver Identification</td>
<td>This is an eight-byte ASCII string without null
termination which identifies the driver and version number
of the Driver Information block. The predefined drivers
supplied with the HDF5 library are identified by the
letters <code>NCSA</code> followed by the first four characters of
the driver name. If the Driver Information block is not
the original version then the last letter(s) of the
identification will be replaced by a version number in
ASCII.
For example, the various versions of the <em>family driver</em>
will be identified by <code>NCSAfami</code>, <code>NCSAfam0</code>,
<code>NCSAfam1</code>, etc.
(<code>NCSAfami</code> is simply <code>NCSAfamily</code> truncated
to eight characters. Subsequent identifiers will be created by
substituting sequential numerical values for the final character,
starting with zero.)
<p>
Identification for user-defined drivers
is arbitrary but should be unique.</td>
</tr>
<tr valign=top>
<td>Driver Information</td>
<td>Driver information is stored in a format defined by the
file driver and encoded/decoded by the driver callbacks
invoked from the <code>H5FD_sb_encode</code> and
<code>H5FD_sb_decode</code> functions.</td>
</tr>
</table>
</center>
<br><br>
<br><br>
<h2><a name="Group">
Disk Format: Level 1 - File Infrastructure</a></h2>
<h3><a name="Btrees">Disk Format: Level 1A - B-link Trees and B-tree Nodes</a></h3>
<p>B-link trees allow flexible storage for objects which tend to grow
in ways that cause the object to be stored discontiguously. B-trees
are described in various algorithms books including "Introduction to
Algorithms" by Thomas H. Cormen, Charles E. Leiserson, and Ronald
L. Rivest. The B-link tree, in which the sibling nodes at a
particular level in the tree are stored in a doubly-linked list,
is described in the "Efficient Locking for Concurrent Operations
on B-trees" paper by Phillip Lehman and S. Bing Yao as published
in the <em>ACM Transactions on Database Systems</em>, Vol. 6,
No. 4, December 1981.
<p>The B-link trees implemented by the file format contain one more
key than the number of children. In other words, each child
pointer out of a B-tree node has a left key and a right key.
The pointers out of internal nodes point to sub-trees while
the pointers out of leaf nodes point to symbol nodes and
raw data chunks.
Aside from that difference, internal nodes and leaf nodes
are identical.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>B-tree Nodes</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>Node Signature</td>
<tr align=center>
<td>Node Type</td>
<td>Node Level</td>
<td colspan=2>Entries Used</td>
<tr align=center>
<td colspan=4>Address of Left Sibling</td>
<tr align=center>
<td colspan=4>Address of Right Sibling</td>
<tr align=center>
<td colspan=4>Key 0 (variable size)</td>
<tr align=center>
<td colspan=4>Address of Child 0</td>
<tr align=center>
<td colspan=4>Key 1 (variable size)</td>
<tr align=center>
<td colspan=4>Address of Child 1</td>
<tr align=center>
<td colspan=4>...</td>
<tr align=center>
<td colspan=4>Key 2<em>K</em> (variable size)</td>
<tr align=center>
<td colspan=4>Address of Child 2<em>K</em></td>
<tr align=center>
<td colspan=4>Key 2<em>K</em>+1 (variable size)</td>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Node Signature</td>
<td>The ASCII character string <code>TREE</code> is
used to indicate the
beginning of a B-link tree node. This gives file
consistency checking utilities a better chance of
reconstructing a damaged file.</td>
</tr>
<tr valign=top>
<td>Node Type</td>
<td>Each B-link tree points to a particular type of data.
This field indicates the type of data as well as
implying the maximum degree <em>K</em> of the tree and
the size of each Key field.
<br>
<dl compact>
<dt>0
<dd>This tree points to group nodes.
<dt>1
<dd>This tree points to a new data chunk.
</dl>
</td>
</tr>
<tr valign=top>
<td>Node Level</td>
<td>The node level indicates the level at which this node
appears in the tree (leaf nodes are at level zero). Not
only does the level indicate whether child pointers
point to sub-trees or to data, but it can also be used
to help file consistency checking utilities reconstruct
damanged trees.</td>
</tr>
<tr valign=top>
<td>Entries Used</td>
<td>This determines the number of children to which this
node points. All nodes of a particular type of tree
have the same maximum degree, but most nodes will point
to less than that number of children. The valid child
pointers and keys appear at the beginning of the node
and the unused pointers and keys appear at the end of
the node. The unused pointers and keys have undefined
values.</td>
</tr>
<tr valign=top>
<td>Address of Left Sibling</td>
<td>This is the file address of the left sibling of the
current node relative to the super block. If the current
node is the left-most node at this level then this field
is the undefined address (all bits set).</td>
</tr>
<tr valign=top>
<td>Address of Right Sibling</td>
<td>This is the file address of the right sibling of the
current node relative to the super block. If the current
node is the right-most node at this level then this
field is the undefined address (all bits set).</td>
</tr>
<tr valign=top>
<td>Keys and Child Pointers</td>
<td>Each tree has 2<em>K</em>+1 keys with 2<em>K</em>
child pointers interleaved between the keys. The number
of keys and child pointers actually containing valid
values is determined by the <em>Entries Used</em> field. If
that field is <em>N</em> then the B-link tree contains
<em>N</em> child pointers and <em>N</em>+1 keys.</td>
</tr>
<tr valign=top>
<td>Key</td>
<td>The format and size of the key values is determined by
the type of data to which this tree points. The keys are
ordered and are boundaries for the contents of the child
pointer; that is, the key values represented by child
<em>N</em> fall between Key <em>N</em> and Key
<em>N</em>+1. Whether the interval is open or closed on
each end is determined by the type of data to which the
tree points.
<p>
The format of the key depends on the node type.
For nodes of node type 1, the key is formatted as follows:
<center>
<table>
<tr valign=top align=left>
<td width=40%>Bytes 1-4</td>
<td>Size of chunk in bytes.</td>
<tr valign=top align=left></tr>
<td>Bytes 4-8</td>
<td>Filter mask, a 32-bit bitfield indicating which
filters have been applied to that chunk.</td>
</tr><tr valign=top align=left>
<td><i>N</i> fields of 8 bytes each</td>
<td>A 64-bit index indicating the offset of the
chunk within the dataset where <i>N</i> is the number
of dimensions of the dataset. For example, if
a chunk in a 3-dimensional dataset begins at the
position <code>[5,5,5]</code>, there will be three
such 8-bit indices, each with the value of
<code>5</code>.</td>
</tr>
</table>
</center>
<p>
For nodes of node type 0, the key is formatted as follows:
<center>
<table>
<tr valign=top align=left>
<td width=40%>A single field of <i>Size of Lengths</i>
bytes</td>
<td>Indicates the byte offset into the local heap
for the first object name in the subtree which
that key describes.</td>
</tr>
</table>
</center>
</td>
</tr>
<tr valign=top>
<td>Child Pointers</td>
<td>The tree node contains file addresses of subtrees or
data depending on the node level. Nodes at Level 0 point
to data addresses, either data chunk or group nodes.
Nodes at non-zero levels point to other nodes of the
same B-tree.</td>
</tr>
</table>
</center>
<p>
Each B-tree node looks like this:
<center>
<table>
<tr valign=top align=center>
<td>key[0]</td><td>&nbsp;&nbsp;</td>
<td>child[0]</td><td>&nbsp;&nbsp;</td>
<td>key[1]</td><td>&nbsp;&nbsp;</td>
<td>child[1]</td><td>&nbsp;&nbsp;</td>
<td>key[2]</td><td>&nbsp;&nbsp;</td>
<td>...</td><td>&nbsp;&nbsp;</td>
<td>...</td><td>&nbsp;&nbsp;</td>
<td>key[<i>N</i>-1]</td><td>&nbsp;&nbsp;</td>
<td>child[<i>N</i>-1]</td><td>&nbsp;&nbsp;</td>
<td>key[<i>N</i>]</td>
</tr>
</table>
</center>
where child[<i>i</i>] is a pointer to a sub-tree (at a level
above Level 0) or to data (at Level 0).
Each key[<i>i</i>] describes an <i>item</i> stored by the B-tree
(a chunk or an object of a group node). The range of values
represented by child[<i>i</i>] are indicated by key[<i>i</i>]
and key[<i>i</i>+1].
<p>The following question must next be answered:
"Is the value described by key[<i>i</i>] contained in
child[<i>i</i>-1] or in child[<i>i</i>]?"
The answer depends on the type of tree.
In trees for groups (node type 0) the object described by
key[<i>i</i>] is the greatest object contained in
child[<i>i</i>-1] while in chunk trees (node type 1) the
chunk described by key[<i>i</i>] is the least chunk in
child[<i>i</i>].
<p>That means that key[0] for group trees is sometimes unused;
it points to offset zero in the heap, which is always the
empty string and compares as "less-than" any valid object name.
<p>And key[<i>N</i>] for chunk trees is sometimes unused;
it contains a chunk offset which compares as "greater-than"
any other chunk offset and has a chunk byte size of zero
to indicate that it is not actually allocated.
<h3><a name="SymbolTable">Disk Format: Level 1B - Group and Symbol Nodes</a></h3>
<p>A group is an object internal to the file that allows
arbitrary nesting of objects (including other groups).
A group maps a set of names to a set of file
address relative to the base address. Certain meta data
for an object to which the group points can be duplicated
in the group symbol table in addition to the object header.
<p>An HDF5 object name space can be stored hierarchically by
partitioning the name into components and storing each
component in a group. The group entry for a
non-ultimate component points to the group containing
the next component. The group entry for the last
component points to the object being named.
<p>A group is a collection of group nodes pointed
to by a B-link tree. Each group node contains entries
for one or more symbols. If an attempt is made to add a
symbol to an already full group node containing
2<em>K</em> entries, then the node is split and one node
contains <em>K</em> symbols and the other contains
<em>K</em>+1 symbols.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>Group Node (A Leaf of a B-tree)</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>Node Signature</td>
<tr align=center>
<td>Version Number</td>
<td>Reserved for Future Use</td>
<td colspan=2>Number of Symbols</td>
<tr align=center>
<td colspan=4><br><br>Group Entries<br><br><br></td>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Node Signature</td>
<td>The ASCII character string <code>SNOD</code> is
used to indicate the
beginning of a group node. This gives file
consistency checking utilities a better chance of
reconstructing a damaged file.</td>
</tr>
<tr valign=top>
<td>Version Number</td>
<td>The version number for the group node. This
document describes version 1.</td>
</tr>
<tr valign=top>
<td>Number of Symbols</td>
<td>Although all group nodes have the same length,
most contain fewer than the maximum possible number of
symbol entries. This field indicates how many entries
contain valid data. The valid entries are packed at the
beginning of the group node while the remaining
entries contain undefined values.</td>
</tr>
<tr valign=top>
<td>Group Entries</td>
<td>Each symbol has an entry in the group node.
The format of the entry is described below.</td>
</tr>
</table>
</center>
<h3><a name="SymbolTableEntry">
Disk Format: Level 1C - Group Entry </a></h3>
<p>Each group entry in a group node is designed
to allow for very fast browsing of stored objects.
Toward that design goal, the group entries
include space for caching certain constant meta data from the
object header.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>Group Entry</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4>Name Offset (&lt;size&gt; bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Object Header Address</td>
</tr>
<tr align=center>
<td colspan=4>Cache Type</td>
</tr>
<tr align=center>
<td colspan=4>Reserved</td>
</tr>
<tr align=center>
<td colspan=4><br><br>Scratch-pad Space (16 bytes)<br><br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Name Offset</td>
<td>This is the byte offset into the group local
heap for the name of the object. The name is null
terminated.</td>
</tr>
<tr valign=top>
<td>Object Header Address</td>
<td>Every object has an object header which serves as a
permanent location for the object's meta data. In addition
to appearing in the object header, some meta data can be
cached in the scratch-pad space.</td>
</tr>
<tr valign=top>
<td>Cache Type</td>
<td>The cache type is determined from the object header.
It also determines the format for the scratch-pad space.
<br>
<dl compact>
<dt>0
<dd>No data is cached by the group entry. This
is guaranteed to be the case when an object header
has a link count greater than one.
<dt>1
<dd>Object header meta data is cached in the group
entry. This implies that the group
entry refers to another group.
<dt>2
<dd>The entry is a symbolic link. The first four bytes
of the scratch-pad space are the offset into the local
heap for the link value. The object header address
will be undefined.
<dt><em>N</em>
<dd>Other cache values can be defined later and
libraries that do not understand the new values will
still work properly.
</dl>
</td>
</tr>
<tr valign=top>
<td>Reserved</td>
<td>These four bytes are present so that the scratch-pad
space is aligned on an eight-byte boundary. They are
always set to zero.</td>
</tr>
<tr valign=top>
<td>Scratch-pad Space</td>
<td>This space is used for different purposes, depending
on the value of the Cache Type field. Any meta-data
about a dataset object represented in the scratch-pad
space is duplicated in the object header for that
dataset. This meta data can include the datatype
and the size of the dataspace for a dataset whose datatype
is atomic and whose dataspace is fixed and less than
four dimensions.
Furthermore, no data is cached in the group
entry scratch-pad space if the object header for
the group entry has a link count greater than
one.</td>
</tr>
</table>
</center>
<h4>Format of the Scratch-pad Space</h4>
<p>The group entry scratch-pad space is formatted
according to the value in the Cache Type field.
<p>If the Cache Type field contains the value zero
(<code>0</code>) then no information is
stored in the scratch-pad space.
<p>If the Cache Type field contains the value one
(<code>1</code>), then the scratch-pad space
contains cached meta data for another object header
in the following format:
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>Object Header Scratch-pad Format</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>Address of B-tree</td>
<tr align=center>
<td colspan=4>Address of Name Heap</td>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Address of B-tree</td>
<td>This is the file address for the root of the
group's B-tree.</td>
</tr>
<tr valign=top>
<td>Address of Name Heap</td>
<td>This is the file address for the group's local
heap, in which are stored the symbol names.</td>
</tr>
</table>
</center>
<p>If the Cache Type field contains the value two
(<code>2</code>), then the scratch-pad space
contains cached meta data for another symbolic link
in the following format:
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>Symbolic Link Scratch-pad Format</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4>Offset to Link Value</td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Offset to Link Value</td>
<td>The value of a symbolic link (that is, the name of the
thing to which it points) is stored in the local heap.
This field is the 4-byte offset into the local heap for
the start of the link value, which is null terminated.</td>
</tr>
</table>
</center>
<h3><a name="LocalHeap">Disk Format: Level 1D - Local Heaps</a></h3>
<p>A heap is a collection of small heap objects. Objects can be
inserted and removed from the heap at any time.
The address of a heap does not change once the heap is created.
References to objects are stored in the group table;
the names of those objects are stored in the local heap.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Local Heaps</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4>Heap Signature</td>
</tr>
<tr align=center>
<td colspan=4>Reserved (zero)</td>
</tr>
<tr align=center>
<td colspan=4>Data Segment Size</td>
</tr>
<tr align=center>
<td colspan=4>Offset to Head of Free-list (&lt;size&gt; bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Address of Data Segment</td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Heap Signature</td>
<td>The ASCII character string <code>HEAP</code>
is used to indicate the
beginning of a heap. This gives file consistency
checking utilities a better chance of reconstructing a
damaged file.</td>
</tr>
<tr valign=top>
<td>Data Segment Size</td>
<td>The total amount of disk memory allocated for the heap
data. This may be larger than the amount of space
required by the object stored in the heap. The extra
unused space holds a linked list of free blocks.</td>
</tr>
<tr valign=top>
<td>Offset to Head of Free-list</td>
<td>This is the offset within the heap data segment of the
first free block (or all 0xff bytes if there is no free
block). The free block contains &lt;size&gt; bytes that
are the offset of the next free chunk (or all 0xff bytes
if this is the last free chunk) followed by &lt;size&gt;
bytes that store the size of this free chunk.</td>
</tr>
<tr valign=top>
<td>Address of Data Segment</td>
<td>The data segment originally starts immediately after
the heap header, but if the data segment must grow as a
result of adding more objects, then the data segment may
be relocated, in its entirety, to another part of the
file.</td>
</tr>
</table>
</center>
<p>Objects within the heap should be aligned on an 8-byte boundary.
<h3><a name="GlobalHeap">Disk Format: Level 1E - Global Heap</a></h3>
<p>Each HDF5 file has a global heap which stores various types of
information which is typically shared between datasets. The
global heap was designed to satisfy these goals:
<ol type="A">
<li>Repeated access to a heap object must be efficient without
resulting in repeated file I/O requests. Since global heap
objects will typically be shared among several datasets, it is
probable that the object will be accessed repeatedly.
<br><br>
<li>Collections of related global heap objects should result in
fewer and larger I/O requests. For instance, a dataset of
void pointers will have a global heap object for each
pointer. Reading the entire set of void pointer objects
should result in a few large I/O requests instead of one small
I/O request for each object.
<br><br>
<li>It should be possible to remove objects from the global heap
and the resulting file hole should be eligible to be reclaimed
for other uses.
<br><br>
</ol>
<p>The implementation of the heap makes use of the memory
management already available at the file level and combines that
with a new top-level object called a <em>collection</em> to
achieve Goal B. The global heap is the set of all collections.
Each global heap object belongs to exactly one collection and
each collection contains one or more global heap objects. For
the purposes of disk I/O and caching, a collection is treated as
an atomic object.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>A Global Heap Collection</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4>Magic Number</td>
</tr>
<tr align=center>
<td>Version</td>
<td colspan=3>Reserved</td>
</td>
<tr align=center>
<td colspan=4>Collection Size</td>
</tr>
<tr align=center>
<td colspan=4><br>Global Heap Object 1
<i>(described below)</i><br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Global Heap Object 2<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>...<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Global Heap Object <em>N</em><br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Global Heap Object 0 (free space)<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Magic Number</td>
<td>The magic number for global heap collections are the
four bytes <code>G</code>, <code>C</code>, <code>O</code>,
and <code>L</code>.</td>
</tr>
<tr valign=top>
<td>Version</td>
<td>Each collection has its own version number so that new
collections can be added to old files. This document
describes version zero of the collections.
</tr>
<tr valign=top>
<td>Collection Data Size</td>
<td>This is the size in bytes of the entire collection
including this field. The default (and minimum)
collection size is 4096 bytes which is a typical file
system block size and which allows for 170 16-byte heap
objects plus their overhead.</td>
</tr>
<tr valign=top>
<td>Object 1 through <em>N</em></td>
<td>The objects are stored in any order with no
intervening unused space.</td>
</tr>
<tr valign=top>
<td>Object 0</td>
<td>Object 0 (zero), when present, represents the free space in
the collection. Free space always appears at the end of
the collection. If the free space is too small to store
the header for Object 0 (described below) then the
header is implied and the collection contains no free space.
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>Global Heap Object</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=2>Object ID</td>
<td colspan=2>Reference Count</td>
</tr>
<tr align=center>
<td colspan=4>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Object Data Size</td>
</tr>
<tr align=center>
<td colspan=4><br>Object Data<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Object ID</td>
<td>Each object has a unique identification number within a
collection. The identification numbers are chosen so that
new objects have the smallest value possible with the
exception that the identifier <code>0</code> always refers to the
object which represents all free space within the
collection.</td>
</tr>
<tr valign=top>
<td>Reference Count</td>
<td>All heap objects have a reference count field. An
object which is referenced from some other part of the
file will have a positive reference count. The reference
count for Object 0 is always zero.</td>
</tr>
<tr valign=top>
<td>Reserved</td>
<td>Zero padding to align next field on an 8-byte
boundary.</td>
</tr>
<tr valign=top>
<td>Object Size</td> <td>This is the size of the the fields
above plus the object data stored for the object. The
actual storage size is rounded up to a multiple of
eight.</td>
</tr>
<tr valign=top>
<td>Object Data</td>
<td>The object data is treated as a one-dimensional array
of bytes to be interpreted by the caller.</td>
</tr>
</table>
</center>
<h3><a name="FreeSpaceIndex">Disk Format: Level 1F - Free-space Heap</a></h3>
<p>The Free-space Index is a collection of blocks of data,
dispersed throughout the file, which are currently not used by
any file objects.
<p>The super block contains a pointer to root of the free-space description;
that pointer is currently (i.e., in HDF5 Release 1.2) required
to be the undefined address <code>0xfff...ff</code>.
<p>The free-sapce index is not otherwise publicly defined at this time.
<!--
<p>The Free-space Index is a collection of blocks of data,
dispersed throughout the file, which are currently not used by
any file objects. The blocks of data are indexed by a B-tree of
their length within the file.
<p>Each B-tree page is composed of the following entries and
B-tree management information, organized as follows:
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<B>HDF5 Free-space Heap Page</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>Free-space Heap Signature</td>
<tr align=center>
<td colspan=4>B-tree Left-link Offset</td>
<tr align=center>
<td colspan=4><br>Length of Free-block #1<br> <br></td>
<tr align=center>
<td colspan=4><br>Offset of Free-block #1<br> <br></td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4><br>Length of Free-block #n<br> <br></td>
<tr align=center>
<td colspan=4><br>Offset of Free-block #n<br> <br></td>
<tr align=center>
<td colspan=4>"High" Offset</td>
<tr align=center>
<td colspan=4>Right-link Offset</td>
</table>
</center>
<p>
<dl>
<dt> The elements of the free-space heap page are described below:
<dd>
<dl>
<dt>Free-space Heap Signature: (4 bytes)
<dd>The ASCII character string <code>FREE</code>
is used to indicate the
beginning of a free-space heap B-tree page. This gives
file consistency checking utilities a better chance of
reconstructing a damaged file.
<dt>B-tree Left-link Offset: (&lt;offset&gt; bytes)
<dd>This value is used to indicate the offset of all offsets
in the B-link-tree which are smaller than the value of the
offset in entry #1. This value is also used to indicate a
leaf node in the B-link-tree by being set to all ones.
<dt>Length of Free-block #n: (&lt;length&gt; bytes)
<dd>This value indicates the length of an unused block in
the file.
<dt>Offset of Free-block #n: (&lt;offset&gt; bytes)
<dd>This value indicates the offset in the file of an
unused block in the file.
<dt>"High" Offset: (4-bytes)
<dd>This offset is used as the upper bound on offsets
contained within a page when the page has been split.
<dt>Right-link Offset: (&lt;offset&gt; bytes)
<dd>This value is used to indicate the offset of the next
child to the right of the parent of this group
page. When there is no node to the right, this value is
all zeros.
</dl>
</dl>
<p>The algorithms for searching and inserting objects in the
B-tree pages are described fully in the Lehman and Yao paper,
which should be read to provide a full description of the
B-tree's usage.
-->
<br><br>
<br><br>
<h2><a name="DataObject">Disk Format: Level 2 - Data Objects </a></h2>
<p>Data objects contain the real information in the file. These
objects compose the scientific data and other information which
are generally thought of as "data" by the end-user. All the
other information in the file is provided as a framework for
these data objects.
<p>A data object is composed of header information and data
information. The header information contains the information
needed to interpret the data information for the data object as
well as additional "meta-data" or pointers to additional
"meta-data" used to describe or annotate each data object.
<h3><a name="ObjectHeader">
Disk Format: Level 2a - Data Object Headers</a></h3>
<p>The header information of an object is designed to encompass
all the information about an object which would be desired to be
known, except for the data itself. This information includes
the dimensionality, number-type, information about how the data
is stored on disk (in external files, compressed, broken up in
blocks, etc.), as well as other information used by the library
to speed up access to the data objects or maintain a file's
integrity. The header of each object is not necessarily located
immediately prior to the object's data in the file and in fact
may be located in any position in the file.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>Object Headers</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=1 width="25%">Version # of Object Header</td>
<td colspan=1 width="25%">Reserved</td>
<td colspan=2 width="50%">Number of Header Messages</td>
</tr>
<tr align=center>
<td colspan=4>Object Reference Count</td>
</tr>
<tr align=center>
<td colspan=4><br>Total Object Header Size<br><br></td>
</tr>
<tr align=center>
<td colspan=2>Header Message Type #1</td>
<td colspan=2>Size of Header Message Data #1</td>
</tr>
<tr align=center>
<td>Flags</td>
<td colspan=3>Reserved</td>
</tr>
<tr align=center>
<td colspan=4><br>Header Message Data #1<br><br></td>
</tr>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
</tr>
<tr align=center>
<td colspan=2>Header Message Type #n</td>
<td colspan=2>Size of Header Message Data #n</td>
</tr>
<tr align=center>
<td>Flags</td>
<td colspan=3>Reserved</td>
</tr>
<tr align=center>
<td colspan=4><br>Header Message Data #n<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version number of the object header</td>
<td>This value is used to determine the format of the
information in the object header. When the format of the
information in the object header is changed, the version number
is incremented and can be used to determine how the
information in the object header is formatted.</td>
</tr>
<tr valign=top>
<td>Reserved</td>
<td>Always set to zero.</td>
</tr>
<tr valign=top>
<td>Number of header messages</td>
<td>This value determines the number of messages listed in
this object header. This provides a fast way for software
to prepare storage for the messages in the header.</td>
</tr>
<tr valign=top>
<td>Object Reference Count</td>
<td>This value specifies the number of references to this
object within the current file. References to the
data object from external files are not tracked.</td>
</tr>
<tr valign=top>
<td>Total Object Header Size</td>
<td>This value specifies the total number of bytes of header
message data following this length field for the current
message as well as any continuation data located elsewhere
in the file.</td>
</tr>
<tr valign=top>
<td>Header Message Type</td>
<td>The header message type specifies the type of
information included in the header message data following
the type along with a small amount of other information.
Bit 15 of the message type is set if the message is
constant (constant messages cannot be changed since they
may be cached in group entries throughout the
file). The header message types for the pre-defined
header messages will be included in further discussion
below.</td>
</tr>
<tr valign=top>
<td>Size of Header Message Data</td>
<td>This value specifies the number of bytes of header
message data following the header message type and length
information for the current message. The size includes
padding bytes to make the message a multiple of eight
bytes.</td>
</tr>
<tr valign=top>
<td>Flags</td>
<td>This is a bit field with the following definition:
<dl>
<dt><code>0</code>
<dd>If set, the message data is constant. This is used
for messages like the datatype message of a dataset.
<dt><code>1</code>
<dd>If set, the message is stored in the global heap and
the Header Message Data field contains a Shared Object
message and the Size of Header Message Data field
contains the size of that Shared Object message.
<dt><code>2-7</code>
<dd>Reserved
</dl>
</td>
<tr valign=top>
<td>Header Message Data</td>
<td>The format and length of this field is determined by the
header message type and size respectively. Some header
message types do not require any data and this information
can be eliminated by setting the length of the message to
zero. The data is padded with enough zeros to make the
size a multiple of eight.</td>
</tr>
</table>
</center>
<p>The header message types and the message data associated with
them compose the critical "meta-data" about each object. Some
header messages are required for each object while others are
optional. Some optional header messages may also be repeated
several times in the header itself, the requirements and number
of times allowed in the header will be noted in each header
message description below.
<P>The following is a list of currently defined header messages:
<hr>
<h4><a name="NILMessage">Name: NIL</a></h4>
<b>Type: </b>0x0000<br>
<b>Length:</b> varies<br>
<b>Status:</b> Optional, may be repeated.<br>
<b>Purpose and Description:</b> The NIL message is used to
indicate a message
which is to be ignored when reading the header messages for a data object.
[Probably one which has been deleted for some reason.]<br>
<b>Format of Data:</b> Unspecified.<br>
<!-- Delete examples throughout doc
<b>Examples:</b> None.
-->
<hr>
<h4><a name="SimpleDataSpace">Name: Simple Dataspace</a></h4>
<b>Type: </b>0x0001<br>
<b>Length:</b> Varies according to the number of dimensions,
as described in the following table<br>
<b>Status:</b> The <em>Simple Dataspace</em> message is required
and may not be repeated. This message is currently used with
datasets and named dataspaces.<br>
<p>The <em>Simple Dataspace</em> message describes the number
of dimensions and size of each dimension that the data object
has. This message is only used for datasets which have a
simple, rectilinear grid layout; datasets requiring a more
complex layout (irregularly structured or unstructured grids, etc.)
must use the <em>Complex Dataspace</em> message for expressing
the space the dataset inhabits.
<i>(Note: The <em>Complex Dataspace</em> functionality is
not yet implemented (as of HDF5 Release 1.2). It is not described
in this document.)</i>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Simple Dataspace Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td>Version</td>
<td>Dimensionality</td>
<td>Flags</td>
<td>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Dimension Size #1 (&lt;size&gt; bytes)</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4>Dimension Size #n (&lt;size&gt; bytes)</td>
<tr align=center>
<td colspan=4>Dimension Maximum #1 (&lt;size&gt; bytes)</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4>Dimension Maximum #n (&lt;size&gt; bytes)</td>
<tr align=center>
<td colspan=4>Permutation Index #1</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4>Permutation Index #n</td>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version </td>
<td>This value is used to determine the format of the
Simple Dataspace Message. When the format of the
information in the message is changed, the version number
is incremented and can be used to determine how the
information in the object header is formatted.</td>
</tr>
<tr valign=top>
<td>Dimensionality</td>
<td>This value is the number of dimensions that the data
object has.</td>
</tr>
<tr valign=top>
<td>Flags</td>
<td>This field is used to store flags to indicate the
presence of parts of this message. Bit 0 (the least
significant bit) is used to indicate that maximum
dimensions are present. Bit 1 is used to indicate that
permutation indices are present for each dimension.</td>
</tr>
<tr valign=top>
<td>Dimension Size #n (&lt;size&gt; bytes)</td>
<td>This value is the current size of the dimension of the
data as stored in the file. The first dimension stored in
the list of dimensions is the slowest changing dimension
and the last dimension stored is the fastest changing
dimension.</td>
</tr>
<tr valign=top>
<td>Dimension Maximum #n (&lt;size&gt; bytes)</td>
<td>This value is the maximum size of the dimension of the
data as stored in the file. This value may be the special
value &lt;UNLIMITED&gt; (all bits set) which indicates
that the data may expand along this dimension
indefinitely. If these values are not stored, the maximum
value of each dimension is assumed to be the same as the
current size value.</td>
</tr>
<tr valign=top>
<td>Permutation Index #n (4 bytes)</td>
<td>This value is the index permutation used to map
each dimension from the canonical representation to an
alternate axis for each dimension. If these values are
not stored, the first dimension stored in the list of
dimensions is the slowest changing dimension and the last
dimension stored is the fastest changing dimension.</td>
</tr>
</table>
</center>
<!-- Delete examples throughout doc
<h4>Examples</h4>
<dl>
<dt> Example #1
<dd>A sample 640 horizontally by 480 vertically raster image
dimension header. The number of dimensions would be set to 2
and the first dimension's size and maximum would both be set
to 480. The second dimension's size and maximum would both be
set to 640
.
<dt>Example #2
<dd>A sample 4 dimensional scientific dataset which is composed
of 30x24x3 slabs of data being written out in an unlimited
series every several minutes as timestep data (currently there
are five slabs). The number of dimensions is 4. The first
dimension size is 5 and its maximum is &lt;UNLIMITED&gt;. The
second through fourth dimension's size and maximum value are
set to 3, 24, and 30 respectively.
<dt>Example #3
<dd>A sample unlimited length text string, currently of length
83. The number of dimensions is 1, the size of the first
dimension is 83 and the maximum of the first dimension is set
to &lt;UNLIMITED&gt;, allowing further text data to be
appended to the string or possibly the string to be replaced
with another string of a different size. (This could also be
stored as a scalar dataset with number-type set to "string")
</dl>
-->
<!-- DELETE ENTIRE DATASPACE SECTION -->
<!--
<hr>
<h4><a name="DataSpaceMessage">Name: Complex Dataspace (Fiber Bundle?)</a></h4>
<b>Type: </b>0x0002<br>
<b>Length:</b> varies<br>
<b>Status:</b> One of the <em>Simple Dataspace</em> or
<em>Complex Dataspace</em> messages is required (but not both) and may
not be repeated.<br> <b>Purpose and Description:</b> The
<em>Dataspace</em> message describes space that the dataset is
mapped onto in a more comprehensive way than the <em>Simple
Dimensionality</em> message is capable of handling. The
dataspace of a dataset encompasses the type of coordinate system
used to locate the dataset's elements as well as the structure and
regularity of the coordinate system. The dataspace also
describes the number of dimensions which the dataset inhabits as
well as a possible higher dimensional space in which the dataset
is located within.
<br>
<b>Format of Data:</b>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<B>HDF5 Dataspace Message Layout</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>Mesh Type</td>
<tr align=center>
<td colspan=4>Logical Dimensionality</td>
</table>
</center>
<p>
<dl>
<dt>The elements of the dimensionality message are described below:
<dd>
<dl>
<dt>Mesh Type: (unsigned 32-bit integer)
<dd>This value indicates whether the grid is
polar/spherical/cartesion,
structured/unstructured and regular/irregular. <br>
The mesh type value is broken up as follows: <br>
<P>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<B>HDF5 Mesh-type Layout</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=1>Mesh Embedding</td>
<td colspan=1>Coordinate System</td>
<td colspan=1>Structure</td>
<td colspan=1>Regularity</td>
</table>
</center>
The following are the definitions of mesh-type bytes:
<dl>
<dt>Mesh Embedding
<dd>This value indicates whether the dataset dataspace
is located within
another dataspace or not:
<dl> <dl>
<dt>&lt;STANDALONE&gt;
<dd>The dataset mesh is self-contained and is not
embedded in another mesh.
<dt>&lt;EMBEDDED&gt;
<dd>The dataset's dataspace is located within
another dataspace, as
described in information below.
</dl> </dl>
<dt>Coordinate System
<dd>This value defines the type of coordinate system
used for the mesh:
<dl> <dl>
<dt>&lt;POLAR&gt;
<dd>The last two dimensions are in polar
coordinates, higher dimensions are
cartesian.
<dt>&lt;SPHERICAL&gt;
<dd>The last three dimensions are in spherical
coordinates, higher dimensions
are cartesian.
<dt>&lt;CARTESIAN&gt;
<dd>All dimensions are in cartesian coordinates.
</dl> </dl>
<dt>Structure
<dd>This value defines the locations of the grid-points
on the axes:
<dl> <dl>
<dt>&lt;STRUCTURED&gt;
<dd>All grid-points are on integral, sequential
locations, starting from 0.
<dt>&lt;UNSTRUCTURED&gt;
<dd>Grid-points locations in each dimension are
explicitly defined and
may be of any numeric datatype.
</dl> </dl>
<dt>Regularity
<dd>This value defines the locations of the dataset
points on the grid:
<dl> <dl>
<dt>&lt;REGULAR&gt;
<dd>All dataset elements are located at the
grid-points defined.
<dt>&lt;IRREGULAR&gt;
<dd>Each dataset element has a particular
grid-location defined.
</dl> </dl>
</dl>
<p>The following grid combinations are currently allowed:
<dl> <dl>
<dt>&lt;POLAR-STRUCTURED-REGULAR&gt;
<dt>&lt;SPHERICAL-STRUCTURED-REGULAR&gt;
<dt>&lt;CARTESIAN-STRUCTURED-REGULAR&gt;
<dt>&lt;POLAR-UNSTRUCTURED-REGULAR&gt;
<dt>&lt;SPHERICAL-UNSTRUCTURED-REGULAR&gt;
<dt>&lt;CARTESIAN-UNSTRUCTURED-REGULAR&gt;
<dt>&lt;CARTESIAN-UNSTRUCTURED-IRREGULAR&gt;
</dl> </dl>
All of the above grid types can be embedded within another
dataspace.
<br> <br>
<dt>Logical Dimensionality: (unsigned 32-bit integer)
<dd>This value is the number of dimensions that the dataset occupies.
<P>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<B>HDF5 Dataspace Embedded Dimensionality Information</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>Embedded Dimensionality</td>
<tr align=center>
<td colspan=4>Embedded Dimension Size #1</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4>Embedded Dimension Size #n</td>
<tr align=center>
<td colspan=4>Embedded Origin Location #1</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4>Embedded Origin Location #n</td>
</table>
</center>
<dt>Embedded Dimensionality: (unsigned 32-bit integer)
<dd>This value is the number of dimensions of the space the
dataset is located
within. i.e. a planar dataset located within a 3-D space,
or a 3-D dataset
which is a subset of another 3-D space, etc.
<dt>Embedded Dimension Size: (unsigned 32-bit integer)
<dd>These values are the sizes of the dimensions of the
embedded dataspace
that the dataset is located within.
<dt>Embedded Origin Location: (unsigned 32-bit integer)
<dd>These values comprise the location of the dataset's
origin within the embedded dataspace.
</dl>
</dl>
[Comment: need some way to handle different orientations of the
dataset dataspace
within the embedded dataspace]<br>
<P>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<B>HDF5 Dataspace Structured/Regular Grid Information</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>Logical Dimension Size #1</td>
<tr align=center>
<td colspan=4>Logical Dimension Maximum #1</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4>Logical Dimension Size #n</td>
<tr align=center>
<td colspan=4>Logical Dimension Maximum #n</td>
</table>
</center>
<p>
<dl>
<dt>The elements of the dimensionality message are described below:
<dd>
<dl>
<dt>Logical Dimension Size #n: (unsigned 32-bit integer)
<dd>This value is the current size of the dimension of the
data as stored in
the file. The first dimension stored in the list of
dimensions is the slowest
changing dimension and the last dimension stored is the
fastest changing
dimension.
<dt>Logical Dimension Maximum #n: (unsigned 32-bit integer)
<dd>This value is the maximum size of the dimension of the
data as stored in
the file. This value may be the special value
&lt;UNLIMITED&gt; which
indicates that the data may expand along this dimension
indefinitely.
</dl>
</dl>
<P>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<B>HDF5 Dataspace Structured/Irregular Grid Information</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4># of Grid Points in Dimension #1</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4># of Grid Points in Dimension #n</td>
<tr align=center>
<td colspan=4>Datatype of Grid Point Locations</td>
<tr align=center>
<td colspan=4>Location of Grid Points in Dimension #1</td>
<tr align=center>
<td colspan=4>.<br>.<br>.<br></td>
<tr align=center>
<td colspan=4>Location of Grid Points in Dimension #n</td>
</table>
</center>
<P>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<B>HDF5 Dataspace Unstructured Grid Information</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4># of Grid Points</td>
<tr align=center>
<td colspan=4>Datatype of Grid Point Locations</td>
<tr align=center>
<td colspan=4>Grid Point Locations<br>.<br>.<br></td>
</table>
</center>
<h4><a name="DataSpaceExample">Examples:</a></h4>
Need some good examples, this is complex!
-->
<hr>
<h4><a name="DataTypeMessage">Name: Datatype</a></h4>
<b>Type:</b> 0x0003<br>
<b>Length:</b> variable<br>
<b>Status:</b> One required per dataset or named datatype<br>
<p>The datatype message defines the datatype for each data point
of a dataset. A datatype can describe an atomic type like a
fixed- or floating-point type or a compound type like a C
struct. A datatype does not, however, describe how data points
are combined to produce a dataset. Datatypes are stored on disk
as a datatype message, which is a list of datatype classes and
their associated properties.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Datatype Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td>Type Class and Version</td>
<td colspan=3>Class Bit Field</td>
</tr>
<tr align=center>
<td colspan=4>Size in Bytes (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4><br><br>Properties<br><br><br></td>
</tr>
</table>
</center>
<p>The Class Bit Field and Properties fields vary depending
on the Type Class, which is the low-order four bits of the Type
Class and Version field (the high-order four bits are the
version, which should be set to the value one). The type class
is one of 0 (fixed-point number), 1 (floating-point number),
2 (date and time), 3 (text string), 4 (bit field), 5 (opaque),
6 (compound), 7 (reference), 8 (enumeration), or 9 (variable-length).
The Class Bit Field is zero and the size of the
Properties field is zero except for the cases noted here.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Fixed-point Numbers (Class 0)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0</td>
<td><b>Byte Order.</b> If zero, byte order is little-endian;
otherwise, byte order is big endian.</td>
</tr>
<tr valign=top>
<td>1, 2</td>
<td><b>Padding type.</b> Bit 1 is the lo_pad type and bit 2
is the hi_pad type. If a datum has unused bits at either
end, then the lo_pad or hi_pad bit is copied to those
locations.</td>
</tr>
<tr valign=top>
<td>3</td>
<td><b>Signed.</b> If this bit is set then the fixed-point
number is in 2's complement form.</td>
</tr>
<tr valign=top>
<td>4-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Properties for Fixed-point Numbers (Class 0)</b>
</caption>
<tr align=center>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
</tr>
<tr align=center>
<td colspan=2>Bit Offset</td>
<td colspan=2>Bit Precision</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Floating-point Numbers (Class 1)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0</td>
<td><b>Byte Order.</b> If zero, byte order is little-endian;
otherwise, byte order is big endian.</td>
</tr>
<tr valign=top>
<td>1, 2, 3</td>
<td><b>Padding type.</b> Bit 1 is the low bits pad type, bit 2
is the high bits pad type, and bit 3 is the internal bits
pad type. If a datum has unused bits at either or between
the sign bit, exponent, or mantissa, then the value of bit
1, 2, or 3 is copied to those locations.</td>
</tr>
<tr valign=top>
<td>4-5</td>
<td><b>Normalization.</b> The value can be 0 if there is no
normalization, 1 if the most significant bit of the
mantissa is always set (except for 0.0), and 2 if the most
signficant bit of the mantissa is not stored but is
implied to be set. The value 3 is reserved and will not
appear in this field.</td>
</tr>
<tr valign=top>
<td>6-7</td>
<td>Reserved (zero).</td>
</tr>
<tr valign=top>
<td>8-15</td>
<td><b>Sign.</b> This is the bit position of the sign
bit.</td>
</tr>
<tr valign=top>
<td>16-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Properties for Floating-point Numbers (Class 1)</b>
</caption>
<tr align=center>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
</tr>
<tr align=center>
<td colspan=2>Bit Offset</td>
<td colspan=2>Bit Precision</td>
</tr>
<tr align=center>
<td>Exponent Location</td>
<td>Exponent Size in Bits</td>
<td>Mantissa Location</td>
<td>Mantissa Size in Bits</td>
</tr>
<tr align=center>
<td colspan=4>Exponent Bias</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Strings (Class 3)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0-3</td>
<td><b>Padding type.</b> This four-bit value determines the
type of padding to use for the string. The values are:
<dl>
<dt><code>0</code> Null terminate.
<dd>A zero byte marks the end of the string and is
guaranteed to be present after converting a long
string to a short string. When converting a short
string to a long string the value is padded with
additional null characters as necessary.
<br><br>
<dt><code>1</code> Null pad.
<dd>Null characters are added to the end of the value
during conversions from short values to long values
but conversion in the opposite direction simply
truncates the value.
<br><br>
<dt><code>2</code> Space pad.
<dd>Space characters are added to the end of the value
during conversions from short values to long values
but conversion in the opposite direction simply
truncates the value. This is the Fortran
representation of the string.
<br><br>
<dt><code>3-15</code> Reserved.
<dd>These values are reserved for future use.
</dl>
</tr>
<tr valign=top>
<td>4-7</td>
<td><b>Character Set.</b> The character set to use for
encoding the string. The only character set supported is
the 8-bit ASCII (zero) so no translations have been defined
yet.</td>
</tr>
<tr valign=top>
<td>8-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Bitfield Types (Class 4)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0</td>
<td><b>Byte Order.</b> If zero, byte order is little-endian;
otherwise, byte order is big endian.</td>
</tr>
<tr valign=top>
<td>1, 2</td>
<td><b>Padding type.</b> Bit 1 is the lo_pad type and bit 2
is the hi_pad type. If a datum has unused bits at either
end, then the lo_pad or hi_pad bit is copied to those
locations.</td>
</tr>
<tr valign=top>
<td>3-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Properties for Bitfield Types (Class 4)</b>
</caption>
<tr align=center>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
</tr>
<tr align=center>
<td colspan=2>Bit Offset</td>
<td colspan=2>Bit Precision</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Opaque Types (Class 5)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Properties for Opaque Types (Class 5)</b>
</caption>
<tr align=center>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
</tr>
<tr align=center>
<td colspan=4><br>Null-terminated ASCII Tag<br>
(multiple of 8 bytes)<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Compound Types (Class 6)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0-15</td>
<td><b>Number of Members.</b> This field contains the number
of members defined for the compound datatype. The member
definitions are listed in the Properties field of the data
type message.
</tr>
<tr valign=top>
<td>15-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>The Properties field of a compound datatype is a list of the
member definitions of the compound datatype. The member
definitions appear one after another with no intervening bytes.
The member types are described with a recursive datatype
message.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Properties for Compound Types (Class 6)</b>
</caption>
<tr align=center>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
</tr>
<tr align=center>
<td colspan=4><br><br>Name (null terminated, multiple of
eight bytes)<br><br><br></td>
</tr>
<tr align=center>
<td colspan=4>Byte Offset of Member in Compound Instance</td>
</tr>
<tr align=center>
<td>Dimensionality</td>
<td colspan=3>reserved</td>
</tr>
<tr align=center>
<td colspan=4>Dimension Permutation</td>
</tr>
<tr align=center>
<td colspan=4>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Size of Dimension 0 (required)</td>
</tr>
<tr align=center>
<td colspan=4>Size of Dimension 1 (required)</td>
</tr>
<tr align=center>
<td colspan=4>Size of Dimension 2 (required)</td>
</tr>
<tr align=center>
<td colspan=4>Size of Dimension 3 (required)</td>
</tr>
<tr align=center>
<td colspan=4><br><br>Member Type Message<br><br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Enumeration Types (Class 8)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0-15</td>
<td><b>Number of Members.</b> The number of name/value
pairs defined for the enumeration type.</td>
</tr>
<tr valign=top>
<td>16-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Properties for Enumeration Types (Class 8)</b>
</caption>
<tr align=center>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
</tr>
<tr align=center>
<td colspan=4><br>Parent Type<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Names<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Values<br><br></td>
</tr>
</table>
</center>
<center>
<table border=0 cellpadding=4 width="80%">
<tr align=left valign=top>
<td valign=top width=20%>Parent Type:</td>
<td valign=top>Each enumeration type is based on some parent type,
usually an integer. The information for that parent type is
described recursively by this field.</td>
</tr><tr align=left valign=top>
<td valign=top>Names:</td>
<td valign=top>The name for each name/value pair. Each name is
stored as a null terminated ASCII string in a multiple of
eight bytes. The names are in no particular order.</td>
</tr><tr align=left valign=top>
<td valign=top>Values:</td>
<td valign=top>The list of values in the same order as the names.
The values are packed (no inter-value padding) and the
size of each value is determined by the parent type.</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Bit Field for Variable-length Types (Class 9)</b>
</caption>
<tr align=center>
<th width="10%">Bits</th>
<th width="90%">Meaning</th>
</tr>
<tr valign=top>
<td>0-3</td>
<td><dl><dt><b>Type</b></dt>
<dt>0 Variable-length sequence</dt>
<dd>This variable-length datatype can be of any sequence
of data. Variable-length sequences do not have padding
or character set information.</dd>
<dt>1 Variable-length string</dt>
<dd>This variable-length datatype is composed of a series of
characters. Variable-length strings have padding and
character set information.</dd></dl>
</td>
</tr>
<tr valign=top>
<td>4-7</td>
<td><dl><dt><b>Padding type</b> (variable-length string only)</dt>
<dd>This four-bit value determines the type of padding
used for variable-length strings. The values are the same
as for the string padding type, as follows:</dd>
<dt>0 Null terminate</dt>
<dd>A zero byte marks the end of a string and is guaranteed
to be present after converting a long string to a short
string. When converting a short string to a long string,
the value is padded with additional null characters
as necessary.
<dt>1 Null pad</dt>
<dd>Null characters are added to the end of the value
during conversion from a short string to a longer string.
Conversion from a long string to a shorter string
simply truncates the value.</dd>
<dt>2 Space pad</dt>
<dd>Space characters are added to the end of the value
during conversion from a short string to a longer string.
Conversion from a long string to a shorter string simply
truncates the value.
This is the Fortran representation of the string.
</dd>
<dt>3-15 Reserved</dt>
<dd>These values are reserved for future use.</dd></dl>
</td>
</tr>
<tr valign=top>
<td>8-11</td>
<td><dl><dt><b>Character set</b> (variable-length string only)</dt>
<dd>This four-bit value specifies the character set
to be used for encoding the string.</dd>
<dt>0 8-bit ASCII</dt>
<dd>As of this writing (July 2002, Release 1.4.4),
8-bit ASCII is the only character set supported.
Therefore, no translations have been defined.</dd></dl>
</td>
</tr>
<tr valign=top>
<td>12-23</td>
<td>Reserved (zero).</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Properties for Variable-length Types (Class 9)</b>
</caption>
<tr align=center>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
<th width="25%">Byte</th>
</tr>
<tr align=center>
<td colspan=4><br>Parent Type<br><br></td>
</tr>
</table>
</center>
<center>
<table border=0 cellpadding=4 width="80%">
<tr align=left valign=top>
<td valign=top width=20%>Parent Type:</td>
<td valign=top>Each variable-length type is based on
some parent type. The information for that parent type is
described recursively by this field.</td>
</tr>
</table>
</center>
<p>
<!--
<p>Datatype examples are <a href="Datatypes.html">here</a>.
-->
<hr>
<h4><a name="FillValueMessage">Name: Data Storage - Fill Value</a></h4>
<b>Type:</b> 0x0004<br>
<b>Length:</b> varies<br>
<b>Status:</b> Optional, may not be repeated.<br>
<p>The fill value message stores a single data point value which
is returned to the application when an uninitialized data point
is read from the dataset. The fill value is interpretted with
the same datatype as the dataset. If no fill value message is
present then a fill value of all zero is assumed.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Fill Value Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4>Size (4 bytes)</td>
</tr>
<tr align=center>
<td colspan=4><br>Fill Value<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Size (4 bytes)</td>
<td>This is the size of the Fill Value field in bytes.</td>
</tr>
<tr valign=top>
<td>Fill Value</td>
<td>The fill value. The bytes of the fill value are
interpreted using the same datatype as for the dataset.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="ReservedMessage_0005">Name: Reserved - Not Assigned Yet</a></h4>
<b>Type:</b> 0x0005<br>
<b>Length:</b> N/A<br>
<b>Status:</b> N/A<br>
<hr>
<h4><a name="CompactDataStorageMessage">Name: Data Storage - Compact</a></h4>
<b>Type:</b> 0x0006<br>
<b>Length:</b> varies<br>
<b>Status:</b> Optional, may not be repeated.<br>
<p>This message indicates that the data for the data object is
stored within the current HDF file by including the actual
data as the header data for this message. The data is
stored internally in
the <em>normal format</em>, i.e. in one chunk, uncompressed, etc.
<P>Note that one and only one of the <em>Data Storage</em> headers can be
stored for each data object.
<P><b>Format of Data:</b> The message data is actually composed
of dataset data, so the format will be determined by the dataset
format.
<!-- Delete examples throughout doc
<h4><a name="CompactDataStorageExample">Examples:</a></h4>
[very straightforward]
-->
<hr>
<h4><a name="ExternalFileListMessage">Name: Data Storage -
External Data Files</a></h4>
<b>Type:</b> 0x0007<BR>
<b>Length:</b> varies<BR>
<b>Status:</b> Optional, may not be repeated.<BR>
<p><b>Purpose and Description:</b> The external object message
indicates that the data for an object is stored outside the HDF5
file. The filename of the object is stored as a Universal
Resource Location (URL) of the actual filename containing the
data. An external file list record also contains the byte offset
of the start of the data within the file and the amount of space
reserved in the file for that data.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>External File List Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td>Version</td>
<td colspan=3>Reserved</td>
</tr>
<tr align=center>
<td colspan=2>Allocated Slots</td>
<td colspan=2>Used Slots</td>
</tr>
<tr align=center>
<td colspan=4><br>Heap Address<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Slot Definitions...<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version </td>
<td>This value is used to determine the format of the
External File List Message. When the format of the
information in the message is changed, the version number
is incremented and can be used to determine how the
information in the object header is formatted.</td>
</tr>
<tr valign=top>
<td>Reserved</td>
<td>This field is reserved for future use.</td>
</tr>
<tr valign=top>
<td>Allocated Slots</td>
<td>The total number of slots allocated in the message. Its
value must be at least as large as the value contained in
the Used Slots field.</td>
</tr>
<tr valign=top>
<td>Used Slots</td>
<td>The number of initial slots which contain valid
information. The remaining slots are zero filled.</td>
</tr>
<tr valign=top>
<td>Heap Address</td>
<td>This is the address of a local name heap which contains
the names for the external files. The name at offset zero
in the heap is always the empty string.</td>
</tr>
<tr valign=top>
<td>Slot Definitions</td>
<td>The slot definitions are stored in order according to
the array addresses they represent. If more slots have
been allocated than what has been used then the defined
slots are all at the beginning of the list.</td>
</tr>
</table>
</center>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>External File List Slot</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4><br>Name Offset (&lt;size&gt; bytes)<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>File Offset (&lt;size&gt; bytes)<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Size<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Name Offset (&lt;size&gt; bytes)</td>
<td>The byte offset within the local name heap for the name
of the file. File names are stored as a URL which has a
protocol name, a host name, a port number, and a file
name:
<code><em>protocol</em>:<em>port</em>//<em>host</em>/<em>file</em></code>.
If the protocol is omitted then "file:" is assumed. If
the port number is omitted then a default port for that
protocol is used. If both the protocol and the port
number are omitted then the colon can also be omitted. If
the double slash and host name are omitted then
"localhost" is assumed. The file name is the only
mandatory part, and if the leading slash is missing then
it is relative to the application's current working
directory (the use of relative names is not
recommended).</td>
</tr>
<tr valign=top>
<td>File Offset (&lt;size&gt; bytes)</td>
<td>This is the byte offset to the start of the data in the
specified file. For files that contain data for a single
dataset this will usually be zero.</td>
</tr>
<tr valign=top>
<td>Size</td>
<td>This is the total number of bytes reserved in the
specified file for raw data storage. For a file that
contains exactly one complete dataset which is not
extendable, the size will usually be the exact size of the
dataset. However, by making the size larger one allows
HDF5 to extend the dataset. The size can be set to a value
larger than the entire file since HDF5 will read zeros
past the end of the file without failing.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="LayoutMessage">Name: Data Storage - Layout</a></h4>
<b>Type:</b> 0x0008<BR>
<b>Length:</b> varies<BR>
<b>Status:</b> Required for datasets, may not be repeated.
<p><b>Purpose and Description:</b> Data layout describes how the
elements of a multi-dimensional array are arranged in the linear
address space of the file. Two types of data layout are
supported:
<ol>
<li>The array can be stored in one contiguous area of the file.
The layout requires that the size of the array be constant and
does not permit chunking, compression, checksums, encryption,
etc. The message stores the total size of the array and the
offset of an element from the beginning of the storage area is
computed as in C.
<li>The array domain can be regularly decomposed into chunks and
each chunk is allocated separately. This layout supports
arbitrary element traversals, compression, encryption, and
checksums, and the chunks can be distributed across external
raw data files (these features are described in other
messages). The message stores the size of a chunk instead of
the size of the entire array; the size of the entire array can
be calculated by traversing the B-tree that stores the chunk
addresses.
</ol>
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<B>Data Layout Message</B>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td>Version</td>
<td>Dimensionality</td>
<td>Layout Class</td>
<td>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Reserved</td>
</tr>
<tr align=center>
<td colspan=4><br>Address<br><br></td>
</tr>
<tr align=center>
<td colspan=4>Dimension 0 (4-bytes)</td>
</tr>
<tr align=center>
<td colspan=4>Dimension 1 (4-bytes)</td>
</tr>
<tr align=center>
<td colspan=4>...</td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version</td>
<td>A version number for the layout message. This
documentation describes version one.</td>
</tr>
<tr valign=top>
<td>Dimensionality</td>
<td>An array has a fixed dimensionality. This field
specifies the number of dimension size fields later in the
message.</td>
</tr>
<tr valign=top>
<td>Layout Class</td>
<td>The layout class specifies how the other fields of the
layout message are to be interpreted. A value of one
indicates contiguous storage while a value of two
indicates chunked storage. Other values will be defined
in the future.</td>
</tr>
<tr valign=top>
<td>Address</td>
<td>For contiguous storage, this is the address of the first
byte of storage. For chunked storage this is the address
of the B-tree that is used to look up the addresses of the
chunks.</td>
</tr>
<tr valign=top>
<td>Dimensions</td>
<td>For contiguous storage the dimensions define the entire
size of the array while for chunked storage they define
the size of a single chunk.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="ReservedMessage_0009">Name: Reserved - Not Assigned Yet</a></h4>
<b>Type:</b> 0x0009<BR>
<b>Length:</b> N/A<BR>
<b>Status:</b> N/A<BR>
<b>Purpose and Description:</b> N/A<BR>
<b>Format of Data:</b> N/A
<hr>
<h4><a name="ReservedMessage_000A">Name: Reserved - Not Assigned Yet</a></h4>
<b>Type:</b> 0x000A<BR>
<b>Length:</b> N/A<BR>
<b>Status:</b> N/A<BR>
<b>Purpose and Description:</b> N/A<BR>
<b>Format of Data:</b> N/A
<hr>
<h4><a name="FilterMessage">Name: Data Storage - Filter Pipeline</a></h4>
<b>Type:</b> 0x000B<BR>
<b>Length:</b> varies<BR>
<b>Status:</b> Optional, may not be repeated.
<p><b>Purpose and Description:</b> This message describes the
filter pipeline which should be applied to the data stream by
providing filter identification numbers, flags, a name, an
client data.
<p>
<center>
<table border align=center cellpadding=4 witdh="80%">
<caption align=top>
<b>Filter Pipeline Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td>Version</td>
<td>Number of Filters</td>
<td colspan=2>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Reserved</td>
</tr>
<tr align=center>
<td colspan=4><br>Filter List<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version</td>
<td>The version number for this message. This document
describes version one.</td>
</tr>
<tr valign=top>
<td>Number of Filters</td>
<td>The total number of filters described by this
message. The maximum possible number of filters in a
message is 32.</td>
</tr>
<tr valign=top>
<td>Filter List</td>
<td>A description of each filter. A filter description
appears in the next table.</td>
</tr>
</table>
</center>
<p>
<center>
<table border align=center cellpadding=4 witdh="80%">
<caption align=top>
<b>Filter Pipeline Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=2>Filter Identification</td>
<td colspan=2>Name Length</td>
</tr>
<tr align=center>
<td colspan=2>Flags</td>
<td colspan=2>Client Data Number of Values</td>
</tr>
<tr align=center>
<td colspan=4><br>Name<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Client Data<br><br></td>
</tr>
<tr align=center>
<td colspan=4>Padding</td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Filter Identification</td>
<td>This is a unique (except in the case of testing)
identifier for the filter. Values from zero through 255
are reserved for filters defined by the NCSA HDF5
library. Values 256 through 511 have been set aside for
use when developing/testing new filters. The remaining
values are allocated to specific filters by contacting the
<a href="mailto:hdf5dev@ncsa.uiuc.edu">HDF5 Development
Team</a>.</td>
</tr>
<tr valign=top>
<td>Name Length</td>
<td>Each filter has an optional null-terminated ASCII name
and this field holds the length of the name including the
null termination padded with nulls to be a multiple of
eight. If the filter has no name then a value of zero is
stored in this field.</td>
</tr>
<tr valign=top>
<td>Flags</td>
<td>The flags indicate certain properties for a filter. The
bit values defined so far are:
<dl>
<dt><code>bit 1</code>
<dd>If set then the filter is an optional filter.
During output, if an optional filter fails it will be
silently removed from the pipeline.
</dl>
</tr>
<tr valign=top>
<td>Client Data Number of Values</td>
<td>Each filter can store a few integer values to control
how the filter operates. The number of entries in the
Client Data array is stored in this field.</td>
</tr>
<tr valign=top>
<td>Name</td>
<td>If the Name Length field is non-zero then it will
contain the size of this field, a multiple of eight. This
field contains a null-terminated, ASCII character
string to serve as a comment/name for the filter.</td>
</tr>
<tr valign=top>
<td>Client Data</td>
<td>This is an array of four-byte integers which will be
passed to the filter function. The Client Data Number of
Values determines the number of elements in the
array.</td>
</tr>
<tr valign=top>
<td>Padding</td>
<td>Four bytes of zeros are added to the message at this
point if the Client Data Number of Values field contains
an odd number.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="AttributeMessage">Name: Attribute</a></h4>
<b>Type:</b> 0x000C<BR>
<b>Length:</b> varies<BR>
<b>Status:</b> Optional, may be repeated.<BR>
<p><b>Purpose and Description:</b> The <em>Attribute</em>
message is used to list objects in the HDF file which are used
as attributes, or "meta-data" about the current object. An
attribute is a small dataset; it has a name, a datatype, a data
space, and raw data. Since attributes are stored in the object
header they must be relatively small (<64kb) and can be
associated with any type of object which has an object header
(groups, datasets, named types and spaces, etc.).
<p>
<center>
<table border align=center cellpadding=4 width="80%">
<caption align=top>
<b>Attribute Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td>Version</td>
<td>Reserved</td>
<td colspan=2>Name Size</td>
</tr>
<tr align=center>
<td colspan=2>Type Size</td>
<td colspan=2>Space Size</td>
</tr>
<tr align=center>
<td colspan=4><br>Name<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Type<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Space<br><br></td>
</tr>
<tr align=center>
<td colspan=4><br>Data<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version</td>
<td>Version number for the message. This document describes
version 1 of attribute messages.</td>
</tr>
<tr valign=top>
<td>Reserved</td>
<td>This field is reserved for later use and is set to
zero.</td>
</tr>
<tr valign=top>
<td>Name Size</td>
<td>The length of the attribute name in bytes including the
null terminator. Note that the Name field below may
contain additional padding not represented by this
field.</td>
</tr>
<tr valign=top>
<td>Type Size</td>
<td>The length of the datatype description in the Type
field below. Note that the Type field may contain
additional padding not represented by this field.</td>
</tr>
<tr valign=top>
<td>Space Size</td>
<td>The length of the dataspace description in the Space
field below. Note that the Space field may contain
additional padding not represented by this field.</td>
</tr>
<tr valign=top>
<td>Name</td>
<td>The null-terminated attribute name. This field is
padded with additional null characters to make it a
multiple of eight bytes.</td>
</tr>
<tr valign=top>
<td>Type</td>
<td>The datatype description follows the same format as
described for the datatype object header message. This
field is padded with additional zero bytes to make it a
multiple of eight bytes.</td>
</tr>
<tr valign=top>
<td>Space</td>
<td>The dataspace description follows the same format as
described for the dataspace object header message. This
field is padded with additional zero bytes to make it a
multiple of eight bytes.</td>
</tr>
<tr valign=top>
<td>Data</td>
<td>The raw data for the attribute. The size is determined
from the datatype and dataspace descriptions. This
field is <em>not</em> padded with additional zero
bytes.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="NameMessage">Name: Object Name</a></h4>
<p><b>Type:</b> 0x000D<br>
<b>Length:</b> varies<br>
<b>Status:</b> Optional, may not be repeated.
<p><b>Purpose and Description:</b> The object name or comment is
designed to be a short description of an object. An object name
is a sequence of non-zero (<code>\0</code>) ASCII characters with no other
formatting included by the library.
<p>
<center>
<table border align=center cellpadding=4 width="80%">
<caption align=top>
<b>Name Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4><br>Name<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Name</td>
<td>A null terminated ASCII character string.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="ModifiedMessage">Name: Object Modification Date &amp; Time</a></h4>
<p><b>Type:</b> 0x000E<br>
<b>Length:</b> fixed<br>
<b>Status:</b> Optional, may not be repeated.
<p><b>Purpose and Description:</b> The object modification date
and time is a timestamp which indicates (using ISO-8601 date and
time format) the last modification of an object. The time is
updated when any object header message changes according to the
system clock where the change was posted.
<p>
<center>
<table border align=center cellpadding=4 width="80%">
<caption align=top>
<b>Modification Time Message</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
</tr>
<tr align=center>
<td colspan=4>Year</td>
</tr>
<tr align=center>
<td colspan=2>Month</td>
<td colspan=2>Day of Month</td>
</tr>
<tr align=center>
<td colspan=2>Hour</td>
<td colspan=2>Minute</td>
</tr>
<tr align=center>
<td colspan=2>Second</td>
<td colspan=2>Reserved</td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Year</td>
<td>The four-digit year as an ASCII string. For example,
<code>1998</code>. All fields of this message should be interpreted
as coordinated universal time (UTC)</td>
</tr>
<tr valign=top>
<td>Month</td>
<td>The month number as a two digit ASCII string where
January is <code>01</code> and December is <code>12</code>.</td>
</tr>
<tr valign=top>
<td>Day of Month</td>
<td>The day number within the month as a two digit ASCII
string. The first day of the month is <code>01</code>.</td>
</tr>
<tr valign=top>
<td>Hour</td>
<td>The hour of the day as a two digit ASCII string where
midnight is <code>00</code> and 11:00pm is <code>23</code>.</td>
</tr>
<tr valign=top>
<td>Minute</td>
<td>The minute of the hour as a two digit ASCII string where
the first minute of the hour is <code>00</code> and
the last is <code>59</code>.</td>
</tr>
<tr valign=top>
<td>Second</td>
<td>The second of the minute as a two digit ASCII string
where the first second of the minute is <code>00</code>
and the last is <code>59</code>.</td>
</tr>
<tr valign=top>
<td>Reserved</td>
<td>This field is reserved and should always be zero.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="SharedMessage">Name: Shared Object Message</a></h4>
<b>Type:</b> 0x000F<br>
<b>Length:</b> 4 Bytes<br>
<b>Status:</b> Optional, may be repeated.
<p>A constant message can be shared among several object headers
by writing that message in the global heap and having the object
headers all point to it. The pointing is accomplished with a
Shared Object message which is understood directly by the object
header layer of the library. It is also possible to have a
message of one object header point to a message in some other
object header, but care must be exercised to prevent cycles.
<p>If a message is shared, then the message appears in the global
heap and its message ID appears in the Header Message Type
field of the object header. Also, the Flags field in the object
header for that message will have bit two set (the
<code>H5O_FLAG_SHARED</code> bit). The message body in the
object header will be that of a Shared Object message defined
here and not that of the pointed-to message.
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=top>
<b>Shared Message Message</b>
</caption>
<tr align=center>
<th width="25%">byte</td>
<th width="25%">byte</td>
<th width="25%">byte</td>
<th width="25%">byte</td>
</tr>
<tr align=center>
<td>Version</td>
<td>Flags</td>
<td colspan=2>Reserved</td>
</tr>
<tr align=center>
<td colspan=4>Reserved</td>
</tr>
<tr align=center>
<td colspan=4><br>Pointer<br><br></td>
</tr>
</table>
</center>
<p>
<center>
<table align=center width="80%">
<tr>
<th width="30%">Field Name</th>
<th width="70%">Description</th>
</tr>
<tr valign=top>
<td>Version</td>
<td>The version number for the message. This document
describes version one of shared messages.</td>
</tr>
<tr valign=top>
<td>Flags</td>
<td>The Shared Message message points to a message which is
shared among multiple object headers. The Flags field
describes the type of sharing:
<dl>
<dt><code>Bit 0</code>
<dd>If this bit is clear then the actual message is the
first message in some other object header; otherwise
the actual message is stored in the global heap.
<dt><code>Bits 2-7</code>
<dd>Reserved (always zero)
</dl>
</tr>
<tr valign=top>
<td>Pointer</td>
<td>This field points to the actual message. The format of
the pointer depends on the value of the Flags field. If
the actual message is in the global heap then the pointer
is the file address of the global heap collection that
holds the message, and a four-byte index into that
collection. Otherwise the pointer is a group entry
that points to some other object header.</td>
</tr>
</table>
</center>
<hr>
<h4><a name="ContinuationMessage">Name: Object Header Continuation</a></h4>
<b>Type:</b> 0x0010<BR>
<b>Length:</b> fixed<BR>
<b>Status:</b> Optional, may be repeated.<BR>
<b>Purpose and Description:</b> The object header continuation is the location
in the file of more header messages for the current data object. This can be
used when header blocks are large, or likely to change over time.<BR>
<b>Format of Data:</b><p>
The object header continuation is formatted as follows (assuming a 4-byte
length &amp; offset are being used in the current file):
<P>
<center>
<table border cellpadding=4 width=60%>
<caption align=bottom>
<B>HDF5 Object Header Continuation Message Layout</B>
</caption>
<tr align=center>
<th width=25%>byte</th>
<th width=25%>byte</th>
<th width=25%>byte</th>
<th width=25%>byte</th>
<tr align=center>
<td colspan=4>Header Continuation Offset</td>
<tr align=center>
<td colspan=4>Header Continuation Length</td>
</table>
</center>
<P>
<dl>
<dt>The elements of the Header Continuation Message are described below:
<dd>
<dl>
<dt>Header Continuation Offset: (&lt;offset&gt; bytes)
<dd>This value is the offset in bytes from the beginning of the file where the
header continuation information is located.
<dt>Header Continuation Length: (&lt;length&gt; bytes)
<dd>This value is the length in bytes of the header continuation information in
the file.
</dl>
</dl>
<!-- Delete examples throughout doc
<h4><a name="ContinuationExample">Examples:</a></h4>
[straightforward]
-->
<hr>
<h4><a name="SymbolTableMessage">Name: Group Message</a></h4>
<b>Type:</b> 0x0011<BR>
<b>Length:</b> fixed<BR>
<b>Status:</b> Required for groups, may not be repeated.<BR>
<b>Purpose and Description:</b> Each group has a B-tree and a
name heap which are pointed to by this message.<BR>
<b>Format of data:</b>
<p>The group message is formatted as follows:
<p>
<center>
<table border cellpadding=4 width="80%">
<caption align=bottom>
<b>HDF5 Object Header Group Message Layout</b>
</caption>
<tr align=center>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<th width="25%">byte</th>
<tr align=center>
<td colspan=4>B-tree Address</td>
<tr align=center>
<td colspan=4>Heap Address</td>
</table>
</center>
<P>
<dl>
<dt>The elements of the Group Message are described below:
<dd>
<dl>
<dt>B-tree Address (&lt;offset&gt; bytes)
<dd>This value is the offset in bytes from the beginning of the file
where the B-tree is located.
<dt>Heap Address (&lt;offset&gt; bytes)
<dd>This value is the offset in bytes from the beginning of the file
where the group name heap is located.
</dl>
</dl>
<h3><a name="SharedObjectHeader">Disk Format: Level 2b - Shared Data Object Headers</a></h3>
<P>In order to share header messages between several dataset objects, object
header messages may be placed into the global heap. Since these
messages require additional information beyond the basic object header message
information, the format of the shared message is detailed below.
<BR> <BR>
<center>
<table border cellpadding=4 width=60%>
<caption align=bottom>
<B>HDF5 Shared Object Header Message</B>
</caption>
<tr align=center>
<th width=25%>byte</th>
<th width=25%>byte</th>
<th width=25%>byte</th>
<th width=25%>byte</th>
<tr align=center>
<td colspan=4>Reference Count of Shared Header Message</td>
<tr align=center>
<td colspan=4><br> Shared Object Header Message<br> <br></td>
</table>
</center>
<p>
<dl>
<dt> The elements of the shared object header message are described below:
<dd>
<dl>
<dt>Reference Count of Shared Header Message: (32-bit unsigned integer)
<dd>This value is used to keep a count of the number of dataset objects which
refer to this message from their dataset headers. When this count reaches zero,
the shared message header may be removed from the global heap.
<dt>Shared Object Header Message: (various lengths)
<dd>The data stored for the shared object header message is formatted in the
same way as the private object header messages described in the object header
description earlier in this document and begins with the header message Type.
</dl>
</dl>
<h3><a name="DataStorage">Disk Format: Level 2c - Data Object Data Storage</a></h3>
<P>The data for an object is stored separately from the header
information in the file and may not actually be located in the HDF5 file
itself if the header indicates that the data is stored externally. The
information for each record in the object is stored according to the
dimensionality of the object (indicated in the dimensionality header message).
Multi-dimensional data is stored in C order [same as current scheme], i.e. the
"last" dimension changes fastest.
<P>Data whose elements are composed of simple number-types are stored in
native-endian IEEE format, unless they are specifically defined as being stored
in a different machine format with the architecture-type information from the
number-type header message. This means that each architecture will need to
[potentially] byte-swap data values into the internal representation for that
particular machine.
<P> Data with a "variable" sized number-type is stored in a data heap
internal to the HDF5 file. Global heap identifiers are stored in the
data object storage.
<P>Data whose elements are composed of pointer number-types are stored in several
different ways depending on the particular pointer type involved. Simple
pointers are just stored as the dataset offset of the object being pointed to with the
size of the pointer being the same number of bytes as offsets in the file.
Partial-object pointers are stored as a heap-ID which points to the following
information within the file-heap: an offset of the object pointed to, number-type
information (same format as header message), dimensionality information (same
format as header message), sub-set start and end information (i.e. a coordinate
location for each), and field start and end names (i.e. a [pointer to the]
string indicating the first field included and a [pointer to the] string name
for the last field).
<P>Data of a compound datatype is stored as a contiguous stream of the items
in the structure, with each item formatted according to its datatype.
</body>
</html>
doxygen/examples/H5.format.1.1.html
+6439
View File
File diff suppressed because it is too large Load Diff
doxygen/examples/H5.format.2.0.html
+14902
View File
File diff suppressed because it is too large Load Diff
doxygen/examples/H5.format.html
+20400
View File
File diff suppressed because it is too large Load Diff
doxygen/examples/H5A_examples.c
+145
View File
@@ -0,0 +1,145 @@
/* -*- c-file-style: "stroustrup" -*- */
#include "hdf5.h"
#include <stdio.h>
#include <stdlib.h>
int
main(void)
{
int ret_val = EXIT_SUCCESS;
//! <!-- [create] -->
{
__label__ fail_acpl, fail_attr, fail_file;
hid_t file, acpl, fspace, attr;
unsigned mode = H5F_ACC_TRUNC;
char file_name[] = "f1.h5";
// attribute names can be arbitrary Unicode strings
char attr_name[] = "Χαρακτηριστικό";
if ((file = H5Fcreate(file_name, mode, H5P_DEFAULT, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
if ((acpl = H5Pcreate(H5P_ATTRIBUTE_CREATE)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_acpl;
}
// use UTF-8 encoding for the attribute name
if (H5Pset_char_encoding(acpl, H5T_CSET_UTF8) < 0) {
ret_val = EXIT_FAILURE;
goto fail_fspace;
}
// create a scalar (singleton) attribute
if ((fspace = H5Screate(H5S_SCALAR)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_fspace;
}
// create an attribute on the root group
if ((attr = H5Acreate2(file, attr_name, H5T_STD_I32LE, fspace, acpl, H5P_DEFAULT)) ==
H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_attr;
}
H5Aclose(attr);
fail_attr:
H5Sclose(fspace);
fail_fspace:
H5Pclose(acpl);
fail_acpl:
H5Fclose(file);
fail_file:;
}
//! <!-- [create] -->
//! <!-- [read] -->
{
__label__ fail_attr, fail_file;
hid_t file, attr;
unsigned mode = H5F_ACC_RDONLY;
char file_name[] = "f1.h5";
char attr_name[] = "Χαρακτηριστικό";
int value;
if ((file = H5Fopen(file_name, mode, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
if ((attr = H5Aopen(file, attr_name, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_attr;
}
// read the attribute value
if (H5Aread(attr, H5T_NATIVE_INT, &value) < 0)
ret_val = EXIT_FAILURE;
// do something w/ the attribute value
H5Aclose(attr);
fail_attr:
H5Fclose(file);
fail_file:;
}
//! <!-- [read] -->
//! <!-- [update] -->
{
__label__ fail_attr, fail_file;
hid_t file, attr;
unsigned mode = H5F_ACC_RDWR;
char file_name[] = "f1.h5";
char attr_name[] = "Χαρακτηριστικό";
int value = 1234;
if ((file = H5Fopen(file_name, mode, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
if ((attr = H5Aopen(file, attr_name, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_attr;
}
// update the attribute value
if (H5Awrite(attr, H5T_NATIVE_INT, &value) < 0)
ret_val = EXIT_FAILURE;
H5Aclose(attr);
fail_attr:
H5Fclose(file);
fail_file:;
}
//! <!-- [update] -->
//! <!-- [delete] -->
{
__label__ fail_attr, fail_file;
hid_t file;
unsigned mode = H5F_ACC_RDWR;
char file_name[] = "f1.h5";
char attr_name[] = "Χαρακτηριστικό";
if ((file = H5Fopen(file_name, mode, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
// delete the attribute
if (H5Adelete(file, attr_name) < 0) {
ret_val = EXIT_FAILURE;
goto fail_attr;
}
fail_attr:
H5Fclose(file);
fail_file:;
}
//! <!-- [delete] -->
return ret_val;
}
doxygen/examples/H5D_examples.c
+173
View File
@@ -0,0 +1,173 @@
/* -*- c-file-style: "stroustrup" -*- */
#include "hdf5.h"
#include <stdio.h>
#include <stdlib.h>
int
main(void)
{
int ret_val = EXIT_SUCCESS;
//! <!-- [create] -->
{
__label__ fail_lcpl, fail_dset, fail_file;
hid_t file, lcpl, fspace, dset;
unsigned mode = H5F_ACC_TRUNC;
char file_name[] = "d1.h5";
// link names can be arbitrary Unicode strings
char dset_name[] = "σύνολο/δεδομένων";
if ((file = H5Fcreate(file_name, mode, H5P_DEFAULT, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
if ((lcpl = H5Pcreate(H5P_LINK_CREATE)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_lcpl;
}
// use UTF-8 encoding for link names
if (H5Pset_char_encoding(lcpl, H5T_CSET_UTF8) < 0) {
ret_val = EXIT_FAILURE;
goto fail_fspace;
}
// create intermediate groups as needed
if (H5Pset_create_intermediate_group(lcpl, 1) < 0) {
ret_val = EXIT_FAILURE;
goto fail_fspace;
}
// create a 1D dataspace
if ((fspace = H5Screate_simple(1, (hsize_t[]){10}, NULL)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_fspace;
}
// create a 32-bit integer dataset
if ((dset = H5Dcreate2(file, dset_name, H5T_STD_I32LE, fspace, lcpl, H5P_DEFAULT, H5P_DEFAULT)) ==
H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_dset;
}
H5Dclose(dset);
fail_dset:
H5Sclose(fspace);
fail_fspace:
H5Pclose(lcpl);
fail_lcpl:
H5Fclose(file);
fail_file:;
}
//! <!-- [create] -->
//! <!-- [read] -->
{
__label__ fail_dset, fail_file;
hid_t file, dset;
unsigned mode = H5F_ACC_RDONLY;
char file_name[] = "d1.h5";
// assume a priori knowledge of dataset name and size
char dset_name[] = "σύνολο/δεδομένων";
int elts[10];
if ((file = H5Fopen(file_name, mode, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
if ((dset = H5Dopen2(file, dset_name, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_dset;
}
// read all dataset elements
if (H5Dread(dset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, elts) < 0)
ret_val = EXIT_FAILURE;
// do something w/ the dataset elements
H5Dclose(dset);
fail_dset:
H5Fclose(file);
fail_file:;
}
//! <!-- [read] -->
//! <!-- [update] -->
{
__label__ fail_update, fail_fspace, fail_dset, fail_file;
hid_t file, dset, fspace;
unsigned mode = H5F_ACC_RDWR;
char file_name[] = "d1.h5";
char dset_name[] = "σύνολο/δεδομένων";
int new_elts[6][2] = {{-1, 1}, {-2, 2}, {-3, 3}, {-4, 4}, {-5, 5}, {-6, 6}};
if ((file = H5Fopen(file_name, mode, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
if ((dset = H5Dopen2(file, dset_name, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_dset;
}
// get the dataset's dataspace
if ((fspace = H5Dget_space(dset)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_fspace;
}
// select the first 5 elements in odd positions
if (H5Sselect_hyperslab(fspace, H5S_SELECT_SET, (hsize_t[]){1}, (hsize_t[]){2}, (hsize_t[]){5},
NULL) < 0) {
ret_val = EXIT_FAILURE;
goto fail_update;
}
// (implicitly) select and write the first 5 elements of the second column of NEW_ELTS
if (H5Dwrite(dset, H5T_NATIVE_INT, H5S_ALL, fspace, H5P_DEFAULT, new_elts) < 0)
ret_val = EXIT_FAILURE;
fail_update:
H5Sclose(fspace);
fail_fspace:
H5Dclose(dset);
fail_dset:
H5Fclose(file);
fail_file:;
}
//! <!-- [update] -->
//! <!-- [delete] -->
{
__label__ fail_delete, fail_file;
hid_t file;
unsigned mode = H5F_ACC_RDWR;
char file_name[] = "d1.h5";
char group_name[] = "σύνολο";
char dset_name[] = "σύνολο/δεδομένων";
if ((file = H5Fopen(file_name, mode, H5P_DEFAULT)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
// delete (unlink) the dataset
if (H5Ldelete(file, dset_name, H5P_DEFAULT) < 0) {
ret_val = EXIT_FAILURE;
goto fail_delete;
}
// the previous call deletes (unlinks) only the dataset
if (H5Ldelete(file, group_name, H5P_DEFAULT) < 0) {
ret_val = EXIT_FAILURE;
goto fail_delete;
}
fail_delete:
H5Fclose(file);
fail_file:;
}
//! <!-- [delete] -->
return ret_val;
}
doxygen/examples/H5F_examples.c
+187
View File
@@ -0,0 +1,187 @@
/* -*- c-file-style: "stroustrup" -*- */
#include "hdf5.h"
#include <stdio.h>
#include <stdlib.h>
int
main(void)
{
int ret_val = EXIT_SUCCESS;
//! <!-- [life_cycle] -->
{
__label__ fail_fapl, fail_fcpl, fail_file;
hid_t fcpl, fapl, file;
if ((fcpl = H5Pcreate(H5P_FILE_CREATE)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_fcpl;
}
else {
// adjust the file creation properties
}
if ((fapl = H5Pcreate(H5P_FILE_ACCESS)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_fapl;
}
else {
// adjust the file access properties
}
unsigned mode = H5F_ACC_EXCL;
char name[] = "f1.h5";
if ((file = H5Fcreate(name, mode, fcpl, fapl)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
// do something useful with FILE
H5Fclose(file);
fail_file:
H5Pclose(fapl);
fail_fapl:
H5Pclose(fcpl);
fail_fcpl:;
}
//! <!-- [life_cycle] -->
//! <!-- [life_cycle_w_open] -->
{
__label__ fail_fapl, fail_file;
hid_t fapl, file;
if ((fapl = H5Pcreate(H5P_FILE_ACCESS)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_fapl;
}
else {
// adjust the file access properties
}
unsigned mode = H5F_ACC_RDWR;
char name[] = "f1.h5";
if ((file = H5Fopen(name, mode, fapl)) == H5I_INVALID_HID) {
ret_val = EXIT_FAILURE;
goto fail_file;
}
// do something useful with FILE
H5Fclose(file);
fail_file:
H5Pclose(fapl);
fail_fapl:;
}
//! <!-- [life_cycle_w_open] -->
//! <!-- [minimal] -->
{
unsigned mode = H5F_ACC_TRUNC;
char name[] = "f11.h5";
hid_t file = H5Fcreate(name, mode, H5P_DEFAULT, H5P_DEFAULT);
if (file != H5I_INVALID_HID)
H5Fclose(file);
else
ret_val = EXIT_FAILURE;
}
//! <!-- [minimal] -->
//! <!-- [open] -->
{
unsigned mode = H5F_ACC_RDONLY;
char name[] = "f11.h5";
hid_t file = H5Fopen(name, mode, H5P_DEFAULT);
if (file != H5I_INVALID_HID)
H5Fclose(file);
else
ret_val = EXIT_FAILURE;
}
//! <!-- [open] -->
//! <!-- [flush] -->
{
unsigned mode = H5F_ACC_RDWR;
char name[] = "f11.h5";
hid_t file = H5Fopen(name, mode, H5P_DEFAULT);
if (file != H5I_INVALID_HID) {
int step;
for (step = 0; step < 1000; ++step) {
// do important work & flush every 20 steps
if (step % 20 == 0) {
if (H5Fflush(file, H5F_SCOPE_LOCAL) < 0) {
perror("H5Fflush failed.");
ret_val = EXIT_FAILURE;
break;
}
}
}
if (H5Fclose(file) < 0)
perror("H5Fclose failed.");
}
else
ret_val = EXIT_FAILURE;
}
//! <!-- [flush] -->
//! <!-- [libver_bounds] -->
{
unsigned mode = H5F_ACC_RDWR;
char name[] = "f11.h5";
hid_t file = H5Fopen(name, mode, H5P_DEFAULT);
if (file != H5I_INVALID_HID) {
if (H5Fset_libver_bounds(file, H5F_LIBVER_EARLIEST, H5F_LIBVER_V18) >= 0) {
// object creation will not exceed HDF5 version 1.8.x
}
else
perror("H5Fset_libver_bounds failed.");
if (H5Fclose(file) < 0)
perror("H5Fclose failed.");
}
else
ret_val = EXIT_FAILURE;
}
//! <!-- [libver_bounds] -->
//! <!-- [mount] -->
{
hid_t file = H5Fopen("f11.h5", H5F_ACC_RDWR, H5P_DEFAULT);
if (file != H5I_INVALID_HID) {
hid_t group, child;
if ((group = H5Gcreate1(file, "mount_point", H5P_DEFAULT)) != H5I_INVALID_HID) {
if ((child = H5Fopen("f1.h5", H5F_ACC_RDONLY, H5P_DEFAULT)) != H5I_INVALID_HID) {
if (H5Fmount(group, ".", child, H5P_DEFAULT) >= 0) {
// do something useful w/ the mounted file
}
else {
ret_val = EXIT_FAILURE;
perror("H5Fmount failed.");
}
H5Fclose(child);
}
H5Gclose(group);
}
H5Fclose(file);
}
else
ret_val = EXIT_FAILURE;
}
//! <!-- [mount] -->
return ret_val;
}
doxygen/examples/H5Pget_metadata_read_attempts.1.c
+22
View File
@@ -0,0 +1,22 @@
/* Get a copy of file access property list */
fapl = H5Pcreate(H5P_FILE_ACCESS);
/* Retrieve the # of read attempts from the file access property list */
H5Pget_metadata_read_attempts(fapl, &attempts);
/*
* The value returned in "attempts" will be 1 (default for non-SWMR access).
*/
/* Set the # of read attempts to 20 */
H5Pset_metadata_read_attempts(fapl, 20);
/* Retrieve the # of read attempts from the file access property list */
H5Pget_metadata_read_attempts(fapl, &attempts);
/*
* The value returned in "attempts" will be 20 as set.
*/
/* Close the property list */
H5Pclose(fapl);
doxygen/examples/H5Pget_metadata_read_attempts.2.c
+44
View File
@@ -0,0 +1,44 @@
/* Open the file with SWMR access and default file access property list */
fid = H5Fopen(FILE, (H5F_ACC_RDONLY | H5F_ACC_SWMR_READ), H5P_DEFAULT);
/* Get the file's file access roperty list */
file_fapl = H5Fget_access_plist(fid);
/* Retrieve the # of read attempts from the file's file access property list */
H5Pget_metadata_read_attempts(file_fapl, &attempts);
/*
* The value returned in "attempts" will be 100 (default for SWMR access).
*/
/* Close the property list */
H5Pclose(file_fapl);
/* Close the file */
H5Fclose(fid);
/* Create a copy of file access property list */
fapl = H5Pcreate(H5P_FILE_ACCESS);
/* Set the # of read attempts */
H5Pset_metadata_read_attempts(fapl, 20);
/* Open the file with SWMR access and the non-default file access property list */
fid = H5Fopen(FILE, (H5F_ACC_RDONLY | H5F_ACC_SWMR_READ), fapl);
/* Get the file's file access roperty list */
file_fapl = H5Fget_access_plist(fid);
/* Retrieve the # of read attempts from the file's file access property list */
H5Pget_metadata_read_attempts(file_fapl, &attempts);
/*
* The value returned in "attempts" will be 20.
*/
/* Close the property lists */
H5Pclose(file_fapl);
H5Pclose(fapl);
/* Close the file */
H5Fclose(fid);
doxygen/examples/H5Pget_metadata_read_attempts.3.c
+44
View File
@@ -0,0 +1,44 @@
/* Open the file with non-SWMR access and default file access property list */
fid = H5Fopen(FILE, H5F_ACC_RDONLY, H5P_DEFAULT);
/* Get the file's file access roperty list */
file_fapl = H5Fget_access_plist(fid);
/* Retrieve the # of read attempts from the file's file access property list */
H5Pget_metadata_read_attempts(file_fapl, &attempts);
/*
* The value returned in "attempts" will be 1 (default for non-SWMR access).
*/
/* Close the property list */
H5Pclose(file_fapl);
/* Close the file */
H5Fclose(fid);
/* Create a copy of file access property list */
fapl = H5Pcreate(H5P_FILE_ACCESS);
/* Set the # of read attempts */
H5Pset_metadata_read_attempts(fapl, 20);
/* Open the file with non-SWMR access and the non-default file access property list */
fid = H5Fopen(FILE, H5F_ACC_RDONLY, fapl);
/* Get the file's file access roperty list */
file_fapl = H5Fget_access_plist(fid);
/* Retrieve the # of read attempts from the file's file access property list */
H5Pget_metadata_read_attempts(file_fapl, &attempts);
/*
* The value returned in "attempts" will be 1 (default for non-SWMR access).
*/
/* Close the property lists */
H5Pclose(file_fapl);
H5Pclose(fapl);
/* Close the file */
H5Fclose(fid);
doxygen/examples/H5Pget_object_flush_cb.c
+41
View File
@@ -0,0 +1,41 @@
hid_t fapl_id;
unsigned counter;
H5F_object_flush_t *ret_cb;
unsigned * ret_counter;
/* Create a copy of the file access property list */
fapl_id = H5Pcreate(H5P_FILE_ACCESS);
/* Set up the object flush property values */
/* flush_cb: callback function to invoke when an object flushes (see below) */
/* counter: user data to pass along to the callback function */
H5Pset_object_flush_cb(fapl_id, flush_cb, &counter);
/* Open the file */
file_id = H5Fopen(FILE, H5F_ACC_RDWR, H5P_DEFAULT);
/* Get the file access property list for the file */
fapl = H5Fget_access_plist(file_id);
/* Retrieve the object flush property values for the file */
H5Pget_object_flush_cb(fapl, &ret_cb, &ret_counter);
/* ret_cb will point to flush_cb() */
/* ret_counter will point to counter */
/*
.
.
.
.
.
.
*/
/* The callback function for the object flush property */
static herr_t
flush_cb(hid_t obj_id, void *_udata)
{
unsigned *flush_ct = (unsigned *)_udata;
++(*flush_ct);
return 0;
}
doxygen/examples/H5Pset_metadata_read_attempts.c
+59
View File
@@ -0,0 +1,59 @@
//! [SWMR Access]
/* Create a copy of file access property list */
fapl = H5Pcreate(H5P_FILE_ACCESS);
/* Set the # of read attempts */
H5Pset_metadata_read_attempts(fapl, 20);
/* Open the file with SWMR access and the non-default file access property list */
fid = H5Fopen(FILE, (H5F_ACC_RDONLY | H5F_ACC_SWMR_READ), fapl);
/* Get the file's file access roperty list */
file_fapl = H5Fget_access_plist(fid);
/* Retrieve the # of read attempts from the file's file access property list */
H5Pget_metadata_read_attempts(file_fapl, &attempts);
/*
* The value returned in "attempts" will be 20.
* The library will use 20 as the number of read attempts
* when reading checksummed metadata in the file
*/
/* Close the property list */
H5Pclose(fapl);
H5Pclose(file_fapl);
/* Close the file */
H5Fclose(fid);
//! [SWMR Access]
//! [non-SWMR Access]
/* Create a copy of file access property list */
fapl = H5Pcreate(H5P_FILE_ACCESS);
/* Set the # of read attempts */
H5Pset_metadata_read_attempts(fapl, 20);
/* Open the file with SWMR access and the non-default file access property list */
fid = H5Fopen(FILE, H5F_ACC_RDONLY, fapl);
/* Get the file's file access roperty list */
file_fapl = H5Fget_access_plist(fid);
/* Retrieve the # of read attempts from the file's file access property list */
H5Pget_metadata_read_attempts(file_fapl, &attempts);
/*
* The value returned in "attempts" will be 1 (default for non-SWMR access).
* The library will use 1 as the number of read attempts
* when reading checksummed metadata in the file
*/
/* Close the property lists */
H5Pclose(fapl);
H5Pclose(file_fapl);
/* Close the file */
H5Fclose(fid);
//! [non-SWMR Access]
doxygen/examples/H5Pset_object_flush_cb.c
+41
View File
@@ -0,0 +1,41 @@
hid_t file_id, fapl_id;
hid_t dataset_id, dapl_id;
unsigned counter;
/* Create a copy of the file access property list */
fapl_id = H5Pcreate(H5P_FILE_ACCESS);
/* Set up the object flush property values */
/* flush_cb: callback function to invoke when an object flushes (see below) */
/* counter: user data to pass along to the callback function */
H5Pset_object_flush_cb(fapl_id, flush_cb, &counter);
/* Open the file */
file_id = H5Fopen(FILE, H5F_ACC_RDWR, H5P_DEFAULT);
/* Create a group */
gid = H5Gcreate2(fid, “group”, H5P_DEFAULT, H5P_DEFAULT_H5P_DEFAULT);
/* Open a dataset */
dataset_id = H5Dopen2(file_id, DATASET, H5P_DEFAULT);
/* The flush will invoke flush_cb() with counter */
H5Dflush(dataset_id);
/* counter will be equal to 1 */
/* ... */
/* The flush will invoke flush_cb() with counter */
H5Gflush(gid);
/* counter will be equal to 2 */
/* ... */
/* The callback function for object flush property */
static herr_t
flush_cb(hid_t obj_id, void *_udata)
{
unsigned *flush_ct = (unsigned *)_udata;
++(*flush_ct);
return 0;
}
doxygen/examples/ImageSpec.html
+1203
View File
@@ -0,0 +1,1203 @@
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Mozilla/4.72 [en] (WinNT; U) [Netscape]">
<title>Image Specification</title>
The HDF5 specification defines the standard objects and storage for the
standard HDF5 objects. (For information about the HDF5 library, model and
specification, see the HDF documentation.)&nbsp; This document is an additional
specification do define a standard profile for how to store image data
in HDF5. Image data in HDF5 is stored as HDF5 datasets with standard attributes
to define the properties of the image.
<p>This specification is primarily concerned with two dimensional raster
data similar to HDF4 Raster Images.&nbsp; Specifications for storing other
types of imagery will be covered in other documents.
<p>This specification defines:
<ul>
<li>
Standard storage and attributes for an Image dataset (<a href="#Sect1">Section
1</a>)</li>
<li>
Standard storage and attributes for Palettes (<a href="#sect2">Section
2</a>)</li>
<li>
Standard for associating Palettes with Images. (<a href="#Sect3">Section
3</a>)</li>
</ul>
<h2>
<a NAME="Sect1"></a>1. HDF5 Image Specification</h2>
<h3>
1.1 Overview</h3>
Image data is stored as an HDF5 dataset with values of HDF5 class Integer
or Float.&nbsp; A common example would be a two dimensional dataset, with
elements of class Integer, e.g., a two dimensional array of unsigned 8
bit integers.&nbsp; However, this specification does not limit the dimensions
or number type that may be used for an Image.
<p>The dataset for an image is distinguished from other datasets by giving
it an attribute "CLASS=IMAGE".&nbsp; In addition, the Image dataset may
have an optional attribute "PALETTE" that is an array of object references
for zero or more palettes. The Image dataset may have additional attributes
to describe the image data, as defined in <a href="#Sect1.2">Section 1.2</a>.
<p>A Palette is an HDF5 dataset which contains color map information.&nbsp;
A Pallet dataset has an attribute "CLASS=PALETTE" and other attributes
indicating the type and size of the palette, as defined in <a href="#sect2">Section
2.1</a>.&nbsp; A Palette is an independent object, which can be shared
among several Image datasets.
<h3>
<a NAME="Sect1.2"></a>1.2&nbsp; Image Attributes</h3>
The attributes for the Image are scalars unless otherwise noted.&nbsp;
The length of String valued attributes should be at least the number of
characters. Optionally, String valued attributes may be stored in a String
longer than the minimum, in which case it must be zero terminated or null
padded.&nbsp; "Required" attributes must always be used. "Optional" attributes
must be used when required.
<br>&nbsp;
<h4>
Attributes</h4>
<dl>
<dt>
Attribute name="<b>CLASS</b>" (Required)</dt>
<dd>
This attribute is type H5T_C_S1, with size 5.</dd>
<dd>
For all Images, the value of this attribute is "IMAGE".</dd>
<dd>
</dd>
<dd>
This attribute identifies this data set as intended to be interpreted as
an image that conforms to the specifications on this page.</dd>
</dl>
<dt>
Attribute name="<b>PALETTE</b>"</dt>
<dl>
<dd>
A Image dataset within an HDF5 file may optionally specify an array of
palettes to be viewed with. The dataset will have an attribute field called
"<b>PALETTE</b>" which contains a one-dimensional array of object reference
pointers (HDF5 datatype H5T_STD_REF_OBJ) which refer to palettes in the
file. The palette datasets must conform to the Palette specification in
<a href="#sect2">section
2 below</a>. The first palette in this array will be the default palette
that the data may be viewed with.</dd>
</dl>
<dl>
<dt>
</dt>
<dt>
Attribute name="<b>IMAGE_SUBCLASS</b>"</dt>
<dd>
If present, the value of this attribute indicates the type of Palette that
should be used with the Image.&nbsp; This attribute is a scalar of type
H5T_C_S1, with size according to the string plus one.&nbsp; The values
are:</dd>
<dl>
<dt>
"IMAGE_GRAYSCALE" (length 15)</dt>
<dd>
A grayscale image</dd>
<dt>
"IMAGE_BITMAP" (length 12)</dt>
<dd>
A bit map image</dd>
<dt>
"IMAGE_TRUECOLOR" (length 15)</dt>
<dd>
A truecolor image</dd>
<dt>
"IMAGE_INDEXED" (length 13)</dt>
<dd>
An indexed image</dd>
<dd>
</dd>
</dl>
<dt>
Attribute name="<b>INTERLACE_MODE</b>"</dt>
<dd>
For images with more than one component for each pixel, this optional attribute
specifies the layout of the data. The values are type H5T_C_S1 of length
15. See <a href="#Section1.3">section 1.3</a> for information about the
storage layout for data.</dd>
<dd>
"INTERLACE_PIXEL" (default): the component value for a pixel are contiguous.</dd>
<dd>
"INTERLACE_PLANE": each component is stored as a plane.</dd>
<dt>
</dt>
<dt>
Attribute name="<b>DISPLAY_ORIGIN</b>"</dt>
<dd>
This optional attribute indicates the intended orientation of the data
on a two-dimensional raster display.&nbsp; The value indicates which corner
the pixel at (0, 0) should be viewed.&nbsp; The values are type H5T_C_S1
of length 2. If DISPLAY_ORIGIN is not set, the orientation is undefined.</dd>
<dd>
"UL": (0,0) is at the upper left.</dd>
<dd>
"LL": (0,0) is at the lower left.</dd>
<dd>
"UR": (0,0) is at the upper right.</dd>
<dd>
"LR": (0,0) is at the lower right.</dd>
</dl>
<dt>
Attribute name="<b>IMAGE_WHITE_IS_ZERO</b>"</dt>
<dl>
<dd>
This attribute is of type H5T_NATIVE_UCHAR.&nbsp; 0 = false, 1 = true .&nbsp;
This is used for images with IMAGE_SUBCLASS="IMAGE_GRAYSCALE" or "IMAGE_BITMAP".</dd>
</dl>
<dl>
<dt>
Attribute name="<b>IMAGE_MINMAXRANGE</b>"</dt>
<dd>
If present, this attribute is an array of two numbers, of the same HDF5
datatype as the data.&nbsp; The first element is the minimum value of the
data, and the second is the maximum.&nbsp; This is used for images with
IMAGE_SUBCLASS="IMAGE_GRAYSCALE", "IMAGE_BITMAP" or "IMAGE_INDEXED".</dd>
</dl>
<dt>
Attribute name="<b>IMAGE_BACKGROUNDINDEX</b>"</dt>
<dl>
<dd>
If set, this attribute indicates the index value that should be interpreted
as the "background color".&nbsp; This attribute is HDF5 type H5T_NATIVE_UINT.</dd>
</dl>
<dt>
Attribute name="<b>IMAGE_TRANSPARENCY</b>"</dt>
<dl>
<dd>
If set, this attribute indicates the index value that should be interpreted
as the "transparent color".&nbsp; This attribute is HDF5 type H5T_NATIVE_UINT.&nbsp;
This attribute may not be used for IMAGE_SUBCLASS="IMAGE_TRUE_COLOR".</dd>
</dl>
<dt>
Attribute name="<b>IMAGE_ASPECTRATIO</b>"</dt>
<dl>
<dd>
If set, this attribute indicates the aspect ratio.</dd>
</dl>
<dt>
Attribute name="<b>IMAGE_COLORMODEL</b>"</dt>
<dl>
<dd>
If set, this attribute indicates the color model of Palette that should
be used with the Image.&nbsp; This attribute is of type H5T_C_S1, with
size 3, 4, or 5.&nbsp; The value is one of the color models described in
the Palette specification in <a href="#sect2.2">section 2.2 below</a>.&nbsp;
This attribute may be used only for IMAGE_SUBCLASS="IMAGE_TRUECOLOR" or
"IMAGE_INDEXED".</dd>
</dl>
<dt>
Attribute name="<b>IMAGE_GAMMACORRECTION</b>"</dt>
<dl>
<dd>
If set, this attribute gives the Gamma correction.&nbsp; The attribute
is type H5T_NATIVE_FLOAT.&nbsp; This attribute may be used only for IMAGE_SUBCLASS="IMAGE_TRUECOLOR"
or "IMAGE_INDEXED".</dd>
</dl>
Attribute name="<b>IMAGE_VERSION</b>" (Required)
<dl>
<dd>
This attribute is of type H5T_C_S1, with size corresponding to the length
of the version string.&nbsp; This attribute identifies the version number
of this specification to which it conforms.&nbsp; The current version number
is "1.2".</dd>
<br>&nbsp;
<p>&nbsp;
<br>&nbsp;
<br>&nbsp;
<center><table BORDER=2 BGCOLOR="#FFFFFF" >
<caption><b>Table 1. Attributes of an Image Dataset</b></caption>
<tr>
<td><b>Attribute Name</b></td>
<td><b>(R = Required</b>
<br><b>O= Optional)</b></td>
<td><b>Type</b></td>
<td><b>String Size</b></td>
<td><b>Value</b></td>
</tr>
<tr>
<td>CLASS</td>
<td>R</td>
<td>String</td>
<td>5</td>
<td>"IMAGE"</td>
</tr>
<tr>
<td>PALETTE</td>
<td>O</td>
<td>Array Object References</td>
<td></td>
<td>&lt;references to Palette datasets><sup>1</sup></td>
</tr>
<tr>
<td>IMAGE_SUBCLASS</td>
<td>O<sup>2</sup></td>
<td>String</td>
<td>15,&nbsp;
<br>12,&nbsp;
<br>15,
<br>13</td>
<td>
<dt>
"IMAGE_GRAYSCALE",</dt>
<dt>
"IMAGE_BITMAP",</dt>
<dt>
"IMAGE_TRUECOLOR",</dt>
<dt>
"IMAGE_INDEXED"</dt>
</td>
</tr>
<tr>
<td>INTERLACE_MODE</td>
<td>O<sup>3,6</sup></td>
<td>String</td>
<td>15</td>
<td>The layout of components if more than one component per pixel.</td>
</tr>
<tr>
<td>DISPLAY_ORIGIN</td>
<td>O</td>
<td>String</td>
<td>2</td>
<td>If set, indicates the intended location of the pixel (0,0).</td>
</tr>
<tr>
<td>IMAGE_WHITE_IS_ZERO</td>
<td>O<sup>3,4</sup></td>
<td>Unsigned Integer</td>
<td></td>
<td>0 = false, 1 = true</td>
</tr>
<tr>
<td>IMAGE_MINMAXRANGE</td>
<td>O<sup>3,5</sup></td>
<td>Array [2] &lt;same datatype as data values></td>
<td></td>
<td>The (&lt;minimum>, &lt;maximum>) value of the data.</td>
</tr>
<tr>
<td>IMAGE_BACKGROUNDINDEX</td>
<td>O<sup>3</sup></td>
<td>Unsigned Integer</td>
<td></td>
<td>The index of the background color.</td>
</tr>
<tr>
<td>IMAGE_TRANSPARENCY</td>
<td>O<sup>3,5</sup></td>
<td>Unsigned Integer</td>
<td></td>
<td>The index of the transparent color.</td>
</tr>
<tr>
<td>IMAGE_ASPECTRATIO</td>
<td>O<sup>3,4</sup></td>
<td>Unsigned Integer</td>
<td></td>
<td>The aspect ratio.</td>
</tr>
<tr>
<td>IMAGE_COLORMODEL</td>
<td>O<sup>3,6</sup></td>
<td>String</td>
<td>3, 4, or 5</td>
<td>The color model, as defined below in the Palette specification for
attribute <b>PAL_COLORMODEL</b>.</td>
</tr>
<tr>
<td>IMAGE_GAMMACORRECTION</td>
<td>O<sup>3,6</sup></td>
<td>Float</td>
<td></td>
<td>The gamma correction.</td>
</tr>
<tr>
<td>IMAGE_VERSION</td>
<td>R</td>
<td>String</td>
<td>3</td>
<td>"1.2"</td>
</tr>
</table></center>
<dl><font size=-1>1.&nbsp; The first element of the array is the default
Palette.</font>
<br><font size=-1>2.&nbsp; This attribute is <b>required</b> for images
that use one of the standard color map types listed.</font>
<br><font size=-1>3. This attribute is <b>required</b> if set for the source
image, in the case that the image is translated from another file into
HDF5.</font>
<br><font size=-1>4.&nbsp; This applies to:&nbsp; IMAGE_SUBCLASS="IMAGE_GRAYSCALE"
or "IMAGE_BITMAP".</font>
<br><font size=-1>5.&nbsp; This applies to:&nbsp; IMAGE_SUBCLASS="IMAGE_GRAYSCALE",
"IMAGE_BITMAP", or "IMAGE_INDEXED".</font>
<br><font size=-1>6.&nbsp; This applies to: IMAGE_SUBCLASS="IMAGE_TRUECOLOR",
or "IMAGE_INDEXED".</font></dl>
</dl>
Table 2 summarizes the standard attributes for an Image datasets using
the common sub-classes. R means that the attribute listed on the leftmost
column is Required for the image subclass on the first row, O means that
the attribute is Optional for that subclass and N that the attribute cannot
be applied to that subclass. The two first rows show the only required
attributes
for all subclasses.
<br>&nbsp;
<table BORDER WIDTH="100%" >
<caption><b>Table 2a. Applicability of Attributes to IMAGE sub-classes</b></caption>
<tr>
<td WIDTH="20%"><b>IMAGE_SUBCLASS</b><sup>1</sup></td>
<td WIDTH="20%"><b>IMAGE_GRAYSCALE</b></td>
<td WIDTH="20%"><b>IMAGE_BITMAP</b></td>
</tr>
<tr>
<td WIDTH="20%">CLASS</td>
<td WIDTH="20%">R</td>
<td WIDTH="20%">R</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_VERSION</td>
<td WIDTH="20%">R</td>
<td WIDTH="20%">R</td>
</tr>
<tr>
<td>INTERLACE_MODE</td>
<td>N</td>
<td>N</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_WHITE_IS_ZERO</td>
<td WIDTH="20%">R</td>
<td WIDTH="20%">R</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_MINMAXRANGE</td>
<td WIDTH="20%">O</td>
<td WIDTH="20%">O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_BACKGROUNDINDEX</td>
<td WIDTH="20%">O</td>
<td WIDTH="20%">O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_TRANSPARENCY</td>
<td WIDTH="20%">O</td>
<td WIDTH="20%">O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_ASPECTRATIO</td>
<td WIDTH="20%">O</td>
<td WIDTH="20%">O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_COLORMODEL</td>
<td WIDTH="20%">N</td>
<td WIDTH="20%">N</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_GAMMACORRECTION</td>
<td WIDTH="20%">N</td>
<td WIDTH="20%">N</td>
</tr>
<tr>
<td WIDTH="20%">PALETTE</td>
<td WIDTH="20%">O</td>
<td WIDTH="20%">O</td>
</tr>
<tr>
<td>DISPLAY_ORIGIN</td>
<td>O</td>
<td>O</td>
</tr>
</table>
<blockquote>&nbsp;</blockquote>
<table BORDER WIDTH="100%" >
<caption><b>Table 2b. Applicability of Attributes to IMAGE sub-classes</b></caption>
<tr>
<td WIDTH="20%"><b>IMAGE_SUBCLASS</b></td>
<td WIDTH="20%"><b>IMAGE_TRUECOLOR</b></td>
<td><b>IMAGE_INDEXED</b></td>
</tr>
<tr>
<td WIDTH="20%">CLASS</td>
<td WIDTH="20%">R</td>
<td>R</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_VERSION</td>
<td WIDTH="20%">R</td>
<td>R</td>
</tr>
<tr>
<td>INTERLACE_MODE</td>
<td>R</td>
<td>N</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_WHITE_IS_ZERO</td>
<td WIDTH="20%">N</td>
<td>N</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_MINMAXRANGE</td>
<td WIDTH="20%">N</td>
<td>O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_BACKGROUNDINDEX</td>
<td WIDTH="20%">N</td>
<td>O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_TRANSPARENCY</td>
<td WIDTH="20%">N</td>
<td>O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_ASPECTRATIO</td>
<td WIDTH="20%">O</td>
<td>O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_COLORMODEL</td>
<td WIDTH="20%">O</td>
<td>O</td>
</tr>
<tr>
<td WIDTH="20%">IMAGE_GAMMACORRECTION</td>
<td WIDTH="20%">O</td>
<td>O</td>
</tr>
<tr>
<td WIDTH="20%">PALETTE</td>
<td WIDTH="20%">O</td>
<td>O</td>
</tr>
<tr>
<td>DISPLAY_ORIGIN</td>
<td>O</td>
<td>O</td>
</tr>
</table>
<h3>
<a NAME="Section1.3"></a>1.3 Storage Layout and Properties for Images</h3>
In the case of an image with more than one component per pixel (e.g., Red,
Green, and Blue), the data may be arranged in one of two ways.&nbsp; Following
HDF4 terminology, the data may be interlaced by pixel or by plane, which
should be indicated by the INTERLACE_MODE&nbsp; attribute.&nbsp; In both
cases, the dataset will have a dataspace with three dimensions, height,
width, and components.&nbsp; The interlace modes specify different orders
for the dimensions.
<br>&nbsp;
<table BORDER COLS=2 WIDTH="100%" >
<caption><b>Table 3. Storage of multiple component image data.</b></caption>
<tr>
<td><b>Interlace Mode</b></td>
<td><b>Dimensions in the Dataspace</b></td>
</tr>
<tr>
<td>INTERLACE_PIXEL</td>
<td>[height][width][pixel components]</td>
</tr>
<tr>
<td>INTERLACE_PLANE</td>
<td>[pixel components][height][width]</td>
</tr>
</table>
<p>For example, consider a 5 (rows) by 10 (column) image, with Red, Green,
and Blue components.&nbsp; Each component is an unsigned byte. In HDF5,
the datatype would be declared as an unsigned 8 bit integer.&nbsp; For
pixel interlace, the dataspace would be a three dimensional array, with
dimensions: [10][5][3].&nbsp; For plane interleave, the dataspace would
be three dimensions: [3][10][5].
<p>In the case of images with only one component, the dataspace may be
either a two dimensional array, or a three dimensional array with the third
dimension of size 1.&nbsp; For example, a 5 by 10 image with 8 bit color
indexes would be an HDF5 dataset with type unsigned 8 bit integer.&nbsp;
The dataspace could be either a two dimensional array, with dimensions
[10][5], or three dimensions, with dimensions either [10][5][1] or [1][10][5].
<p>Image datasets may be stored with any chunking or compression properties
supported by HDF5.
<p><b>A note concerning compatibility with HDF5 GR interface: </b>An Image
dataset is stored as an HDF5 dataset.&nbsp; It is important to note that
the order of the dimensions is the same as for any other HDF5 dataset.&nbsp;
For a two dimensional image that is to be stored as a series of horizontal
scan lines, with the scan lines contiguous (i.e., the fastest changing
dimension is 'width'), the image will have a dataspace with <i>dim[0] =
height</i> and <i>dim[1]</i> = <i>width</i>.&nbsp; This is completely consistent
with all other HDF5 datasets.
<p>Users familiar with HDF4 should be cautioned that <i>this is not the
same as HDF4</i>, and specifically is not consistent with what the HDF4
GR interface does.
<br>&nbsp;
<h2>
<a NAME="sect2"></a>2.&nbsp; HDF5 Palette Specification</h2>
<h3>
2.1 Overview</h3>
A palette is the means by which color is applied to an image and is also
referred to as a color lookup table. It is a table in which every row contains
the numerical representation of a particular color. In the example of an
8 bit standard RGB color model palette, this numerical representation of
a color is presented as a triplet specifying the intensity of red, green,
and blue components that make up each color.
<center>
<p><img SRC="Palettes.fm.anc.gif" ></center>
<p>In this example, the color component numeric type is an 8 bit unsigned
integer. While this is most common and recommended for general use, other
component color numeric datatypes, such as a 16 bit unsigned integer ,
may be used. This type is specified as the type attribute of the palette
dataset. (see H5Tget_type(), H5Tset_type())
<p>The minimum and maximum values of the component color numeric are specified
as attribute of the palette dataset. See below (attribute PAL_MINMAXNUMERIC).
If these attributes do not exist, it is assumed that the range of values
will fill the space of the color numeric type. i.e. with an 8 bit unsigned
integer, the valid range would be 0 to 255 for each color component.
<p>The HDF5 palette specification additionally allows for color models
beyond RGB. YUV, HSV, CMY, CMYK, YCbCr color models are supported, and
may be specified as a color model attribute of the palette dataset. <i>(see
"Palette Attributes" for details)</i>.
<p>In HDF 4 and earlier, palettes were limited to 256 colors. The HDF5
palette specification allows for palettes of varying length. The length
is specified as the number of rows of the palette dataset.
<br>&nbsp;
<br>&nbsp;
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#666666" >
<tr>
<td><font color="#FFFFFF">Important Note: The specification of the Indexed
Palette will change substantially in the next version.&nbsp; The Palette
described here is <i>denigrated</i> and is not supported.</font></td>
</tr>
</table>
<br>&nbsp;
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td><i>Denigrated</i>
<p>In a standard palette, the color entries are indexed directly. HDF5
supports the notion of a range index table. Such a table defines an ascending
ordered list of ranges that map dataset values to the palette. If a range
index table exists for the palette, the PAL_TYPE attribute will be set
to "RANGEINDEX", and the PAL_RANGEINDEX attribute will contain an object
reference to a range index table array. If not, the PAL_TYPE attribute
either does not exist, or will be set to "STANDARD".
<p>The range index table array consists of a one dimensional array with
the same length as the palette dataset - 1. Ideally, the range index would
be of the same type as the dataset it refers to, however this is not a
requirement.
<p><b>Example 2: A range index array of type floating point</b>
<center>
<p><img SRC="PaletteExample1.gif" ></center>
<p>The range index array attribute defines the "<i>to</i>" of the range.
Notice that the range index array attribute is one less entry in size than
the palette. The first entry of 0.1259, specifies that all values below
and up to 0.1259 inclusive, will map to the first palette entry. The second
entry signifies that all values greater than 0.1259 up to 0.3278 inclusive,
will map to the second palette entry, etc. All value greater than the last
range index array attribute (100000) map to the last entry in the palette.</td>
</tr>
</table>
<h3>
<a NAME="sect2.2"></a>2.2. Palette Attributes</h3>
A palette exists in an HDF file as an independent data set with accompanying
attributes.&nbsp; The Palette attributes are scalars except where noted
otherwise.&nbsp; String values should have size the length of the string
value plus one.&nbsp; "Required" attributes must be used.&nbsp; "Optional"
attributes must be used when required.
<p>These attributes are defined as follows:
<dl>
<dt>
Attribute name="<b>CLASS</b>" (Required)</dt>
<dd>
This attribute is of type H5T_C_S1, with size 7.</dd>
<dd>
For all palettes, the value of this attribute is "PALETTE". This attribute
identifies this palette data set as a palette that conforms to the specifications
on this page.</dd>
<dt>
Attribute name="<b>PAL_COLORMODEL</b>" (Required)</dt>
<dd>
This attribute is of type H5T_C_S1, with size 3, 4, or 5.</dd>
<dd>
Possible values for this are "RGB", "YUV", "CMY", "CMYK", "YCbCr", "HSV".</dd>
<dd>
This defines the color model that the entries in the palette data set represent.</dd>
<dl>
<dt>
"RGB"</dt>
<dd>
Each color index contains a triplet where the the first value defines the
red component, second defines the green component, and the third the blue
component.</dd>
<dt>
"CMY"</dt>
<dd>
Each color index contains a triplet where the the first value defines the
cyan component, second defines the magenta component, and the third the
yellow component.</dd>
<dt>
"CMYK"</dt>
<dd>
Each color index contains a quadruplet where the the first value defines
the cyan component, second defines the magenta component, the third the
yellow component, and the forth the black component.</dd>
<dt>
"YCbCr"</dt>
<dd>
Class Y encoding model. Each color index contains a triplet where the the
first value defines the luminance, second defines the Cb Chromonance, and
the third the Cr Chromonance.</dd>
<dt>
"YUV"</dt>
<dd>
Composite encoding color model. Each color index contains a triplet where
the the first value defines the luminance component, second defines the
chromonance component, and the third the value component.</dd>
<dt>
"HSV"</dt>
<dd>
Each color index contains a triplet where the the first value defines the
hue component, second defines the saturation component, and the third the
value component. The hue component defines the hue spectrum with a low
value representing magenta/red progressing to a high value which would
represent blue/magenta, passing through yellow, green, cyan. A low value
for the saturation component means less color saturation than a high value.
A low value for <i>value</i> will be darker than a high value.</dd>
<dd>
</dd>
</dl>
<dt>
Attribute name="<b>PAL_TYPE</b>" (Required)</dt>
<dd>
This attribute is of type H5T_C_S1, with size 9 or 10.</dd>
<dd>
The current supported values for this attribute are : "STANDARD8" or "RANGEINDEX"</dd>
<dd>
A PAL_TYPE of "STANDARD8" defines a palette dataset such that the first
entry defines index 0, the second entry defines index 1, etc. up until
the length of the palette - 1. This assumes an image dataset with direct
indexes into the palette.</dd>
</dl>
<dl>&nbsp;
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td><i>Denigrated</i>
<p>If the PAL_TYPE is set to "RANGEINDEX", there will be an additional
attribute with a name of "<b>PAL_RANGEINDEX</b>",&nbsp; (See example 2
for more details)</td>
</tr>
</table>
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>
<dt>
Attribute name="<b>PAL_RANGEINDEX</b>"&nbsp;&nbsp; <i>(Denigrated)</i></dt>
<dl>
<dd>
The <b>PAL_RANGEINDEX</b> attribute contains an HDF object reference (HDF5
datatype H5T_STD_REF_OBJ) pointer which specifies a range index array in
the file to be used for color lookups for the palette.&nbsp; (Only for
PAL_TYPE="RANGEINDEX")</dd>
</dl>
</td>
</tr>
</table>
<dt>
Attribute name="<b>PAL_MINMAXNUMERIC</b>"</dt>
<dl>
<dt>
If present, this attribute is an array of two numbers, of the same HDF5
datatype as the palette elements or color numerics.</dt>
<br>They specify the minimum and maximum values of the color numeric components.
For example, if the palette was an RGB of type Float, the color numeric
range for Red, Green, and Blue could be set to be between 0.0 and 1.0.
The intensity of the color guns would then be scaled accordingly to be
between this minimum and maximum attribute.</dl>
Attribute name="<b>PAL_VERSION</b>"&nbsp; (Required)
<dl>This attribute is of type H5T_C_S1, with size corresponding to the
length of the version string.&nbsp; This attribute identifies the version
number of this specification to which it conforms.&nbsp; The current version
is "1.2".</dl>
<center><table BORDER=2 BGCOLOR="#FFFFFF" >
<caption><b>Table 4. Attributes of a Palette Dataset</b></caption>
<tr>
<td><b>Attribute Name</b></td>
<td><b>(R = Required,</b>
<br><b>O = Optional)</b></td>
<td><b>Type</b></td>
<td><b>String Size</b></td>
<td><b>Value</b></td>
</tr>
<tr>
<td>CLASS</td>
<td>R</td>
<td>String</td>
<td>
<center>7</center>
</td>
<td>"PALETTE"</td>
</tr>
<tr>
<td>PAL_COLORMODEL</td>
<td>R</td>
<td>String</td>
<td>
<center>3, 4, or 5</center>
</td>
<td>Color Model:&nbsp; "RGB", YUV", "CMY", "CMYK", "YCbCr", or "HSV"</td>
</tr>
<tr>
<td>PAL_TYPE</td>
<td>R</td>
<td>String</td>
<td>
<center>9</center>
<p><br>
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>or 10</td>
</tr>
</table>
</td>
<td>"STANDARD8"&nbsp;
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>or "RANGEINDEX" <i>(Denigrated)</i></td>
</tr>
</table>
</td>
</tr>
<tr>
<td>
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td><i>Denigrated</i>
<br>RANGE_INDEX</td>
</tr>
</table>
</td>
<td></td>
<td>
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>Object Reference&nbsp;</td>
</tr>
</table>
</td>
<td></td>
<td>
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>&lt;Object Reference to Dataset of range index values></td>
</tr>
</table>
</td>
</tr>
<tr>
<td>PAL_MINMAXNUMERIC</td>
<td>O</td>
<td>Array[2] of &lt;same datatype as palette></td>
<td></td>
<td>The first value is the &lt;Minimum value for color values>, the second
value is &lt;Maximum value for color values><sup>2</sup></td>
</tr>
<tr>
<td>PAL_VERSION</td>
<td>R</td>
<td>String</td>
<td>4</td>
<td>"1.2"</td>
</tr>
</table></center>
<dl>&nbsp;
<table BORDER COLS=1 WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td><font size=-1>1.&nbsp; The RANGE_INDEX attribute is required if the
PAL_TYPE is "RANGEINDEX".&nbsp; Otherwise, the RANGE_INDEX attribute should
be omitted. (Range index is denigrated.)</font></td>
</tr>
</table>
<font size=-1>2.&nbsp; The minimum and maximum are optional.&nbsp; If not
set, the range is assumed to the maximum range of the number type.&nbsp;
If one of these attributes is set, then both should be set.&nbsp; The value
of the minimum must be less than or equal to the value of the maximum.</font></dl>
</dl>
Table 5 summarized the uses of the standard attributes for a palette dataset.
R means that the attribute listed on the leftmost column is Required for
the palette type on the first row, O means that the attribute is Optional
for that type and N that the attribute cannot be applied to that type.
The four first rows show the attributes that are always required&nbsp;
for the two palette types.
<br>&nbsp;
<br>&nbsp;
<table BORDER WIDTH="100%" >
<caption><b>Table 5. Applicability of Attributes</b></caption>
<tr>
<td WIDTH="33%"><b>PAL_TYPE</b></td>
<td WIDTH="33%"><b>STANDARD8</b></td>
<td WIDTH="34%"><b>RANGEINDEX</b></td>
</tr>
<tr>
<td WIDTH="33%">CLASS</td>
<td WIDTH="33%">R</td>
<td WIDTH="34%">R</td>
</tr>
<tr>
<td WIDTH="33%">PAL_VERSION</td>
<td WIDTH="33%">R</td>
<td WIDTH="34%">R</td>
</tr>
<tr>
<td WIDTH="33%">PAL_COLORMODEL</td>
<td WIDTH="33%">R</td>
<td WIDTH="34%">R</td>
</tr>
<tr>
<td WIDTH="33%">RANGE_INDEX</td>
<td WIDTH="33%">N</td>
<td WIDTH="34%">R</td>
</tr>
<tr>
<td WIDTH="33%">PAL_MINMAXNUMERIC</td>
<td WIDTH="33%">O</td>
<td WIDTH="34%">O</td>
</tr>
</table>
<h3>
2.3. Storage Layout for Palettes</h3>
The values of the Palette are stored as a dataset.&nbsp; The datatype can
be any HDF 5 atomic numeric type.&nbsp; The dataset will have dimensions
(<tt>nentries</tt>&nbsp; by&nbsp; <tt>ncomponents</tt>), where '<tt>nentries</tt>'
is the number of colors (usually 256) and '<tt>ncomponents'</tt> is the
number of values per color (3 for <b>RGB</b>, 4 for <b>CMYK</b>, etc.)
<br>&nbsp;
<h2>
<a NAME="Sect3"></a>3.&nbsp; Consistency and Correlation of Image and Palette
Attributes</h2>
The objects in this specification are an extension to the base HDF5 specification
and library.&nbsp; They are accessible with the standard HDF5 library,
but the semantics of the objects are not enforced by the base library.&nbsp;
For example, it is perfectly possible to add an attribute called <b>IMAGE</b>
to <i>any</i> dataset, or to include an object reference to <i>any</i>
HDF5 dataset in a <b>PALETTE</b> attribute.&nbsp; This would be a valid
HDF5 file, but not conformant to this specification.&nbsp; The rules defined
in this specification must be implemented with appropriate software, and
applications must use conforming software to assure correctness.
<p>The Image and Palette specifications include several redundant standard
attributes, such as the <b>IMAGE_COLORMODEL</b> and the <b>PAL_COLORMODEL</b>.&nbsp;
These attributes are informative not normative, in that it is acceptable
to attach a Palette to an Image dataset even if their attributes do not
match.&nbsp; Software is not required to enforce consistency, and files
may contain mismatched associations of Images and Palettes.&nbsp; In all
cases, it is up to applications to determine what kinds of images and color
models can be supported.
<p>For example, an Image that was created from a file with an "RGB" may
have a "YUV" Palette in its <b>PALETTE</b> attribute array.&nbsp; This
would be a legal HDF5 file and also conforms to this specification, although
it may or may not be correct for a given application.</p>
</body>
</html>
doxygen/examples/PaletteExample1.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.7 KiB

doxygen/examples/Palettes.fm.anc.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.6 KiB

doxygen/examples/TableSpec.html
+193
View File
@@ -0,0 +1,193 @@
<html>
<head>
<title>HDF5 Table Specification</title>
</head>
The HDF5 specification defines the standard objects and storage for the
standard HDF5 objects. (For information about the HDF5 library, model and
specification, see the HDF documentation.)&nbsp; This document is an additional
specification do define a standard profile for how to store tables in HDF5.
Table data in HDF5 is stored as HDF5 datasets with standard attributes to define
the properties of the tables.
<h2>
1. Overview</h2>
A generic table is a sequence of records, each record has a name and a type.
Table data is stored as an HDF5 one dimensional compound dataset.&nbsp; A table
is defined as a collection of records whose values are stored in fixed-length
fields. All records have the same structure and all values in each field have
the same data type.
<p>The dataset for a table is distinguished from other datasets by giving
it an attribute &quot;CLASS=TABLE&quot;.&nbsp;&nbsp;
Optional attributes allow the storage of a title for the Table and for
each column, and a fill value for each column.
<h2>
2.&nbsp; Table Attributes</h2>
The attributes for the Table are strings.&nbsp;They are written with the <a href="RM_H5LT.html#H5LTset_attribute_string"><code>H5LTset_attribute_string</code></a>
Lite API function.&nbsp; "Required" attributes must always be used. "Optional" attributes
must be used when required.
<br>&nbsp;
<h4>
Attributes</h4>
<dl>
<dt>
Attribute name="<b>CLASS</b>" (Required)</dt>
<dd>
This attribute is type H5T_C_S1, with size 5.</dd>
<dd>
For all Tables, the value of this attribute is &quot;TABLE&quot;.</dd>
<dd>
This attribute identifies this data set as intended to be interpreted as Table that conforms to the specifications on this page.</dd>
</dl>
<dl>
Attribute name="<b>VERSION</b>" (Required)
<dd>
This attribute is of type H5T_C_S1, with size corresponding to the length
of the version string.&nbsp; This attribute identifies the version number
of this specification to which it conforms.&nbsp; The current version number
is &quot;0.2&quot;.</dd>
</dl>
<dl>
<dt>
Attribute name="<b>TITLE</b>" (Optional)</dt>
<dd>
The <b>TITLE</b> is an optional String that is to be used as the
informative title of the whole table.
The <b>TITLE</b> is set with the parameter <code> table_title</code> of the function
<a href="RM_H5TB.html#H5TBmake_table"> <code> H5TBmake_table</code></a>.&nbsp;</dd>
</dl>
<dl>
<dt>
Attribute name="<b>FIELD_(n)_NAME</b>" (Required)</dt>
<dd>
The <b>FIELD_(n)_NAME</b> is an optional String that is to be used as the
informative title of column <b>n</b> of the table.
For each of the fields the word FIELD_ is concatenated with
the zero based field (n) index together with the name of the field.</dd>
</dl>
<dl>
<dt>
Attribute name="<b>FIELD_(n)_FILL</b>" (Optional)</dt>
<dd>
The <b>FIELD_(n)_FILL</b> is an optional String that is the fill value for
column <b>n</b> of the table.
For each of the fields the word FIELD_ is concatenated with
the zero based field (n) index together with the fill value, if present.
This value is written only when a fill value is defined for the table.</dd>
</dl>
<dl>
<br>&nbsp;
<center><table BORDER=2 BGCOLOR="#FFFFFF" >
<caption><b>Table 1. Attributes of an Image Dataset</b></caption>
<tr>
<td><b>Attribute Name</b></td>
<td><b>(R = Required</b>
<br><b>O= Optional)</b></td>
<td><b>Type</b></td>
<td><b>String Size</b></td>
<td><b>Value</b></td>
</tr>
<tr>
<td>CLASS</td>
<td>R</td>
<td>String</td>
<td>5</td>
<td>&quot;TABLE&quot;</td>
</tr>
<tr>
<td>VERSION</td>
<td>R</td>
<td>String</td>
<td>3</td>
<td>&quot;0.2&quot;</td>
</tr>
<tr>
<td>TITLE</td>
<td>O</td>
<td>String</td>
<td>&nbsp;</td>
<td>
<tr>
<td>FIELD_(n)_NAME</td>
<td>R</td>
<td>String</td>
<td>&nbsp;</td>
<td>
&nbsp;
<tr>
<td>FIELD_(n)_FILL</td>
<td>O*</td>
<td>String</td>
<td>&nbsp;</td>
<td>
&nbsp;
</table>
</center>
</dl>
<p>
<center>
&nbsp;
</center>
<i>* </i>The attribute FIELD_(n)_FILL is written to the table if a fill value is
specified on the creation of the Table. Otherwise, it is not.<p>The following
section of code shows the calls necessary to the creation of a table.
<p><code>/* Create a new HDF5 file using default properties. */<br>
file_id = H5Fcreate( &quot;my_table.h5&quot;, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT );</code> </p>
<p><code>/* Call the make table function */<br>
</code> <code>H5TBmake_table( "Table Title", file_id, "Table1", NFIELDS, NRECORDS, dst_size,&nbsp;<br>
field_names, dst_offset, field_type,&nbsp;<br>
chunk_size, fill_data, compress, p_data )&nbsp;</code> </p>
<p><code> /* Close the file. */<br>
status = H5Fclose( file_id );</code> </p>
</body>
doxygen/examples/ThreadSafeLibrary.html
+787
View File
@@ -0,0 +1,787 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
<html lang="en-US">
<head>
<title>Thread Safe Library</title>
</head>
<h1>1. Library header files and conditional compilation</h1>
<p>
The following code is placed at the beginning of H5private.h:
</p>
<blockquote>
<pre>
#ifdef H5_HAVE_THREADSAFE
#include &lt;pthread.h&gt;
#endif
</pre>
</blockquote>
<p>
<code>H5_HAVE_THREADSAFE</code> is defined when the HDF-5 library is
compiled with the --enable-threadsafe configuration option. In general,
code for the non-threadsafe version of HDF-5 library are placed within
the <code>#else</code> part of the conditional compilation. The exception
to this rule are the changes to the <code>FUNC_ENTER</code> (in
H5private.h), <code>HRETURN</code> and <code>HRETURN_ERROR</code> (in
H5Eprivate.h) macros (see section 3.2).
</p>
<h1>2. Global variables/structures</h1>
<h2>2.1 Global library initialization variable</h2>
<p>
In the threadsafe implementation, the global library initialization
variable <code>H5_libinit_g</code> is changed to a global structure
consisting of the variable with its associated lock (locks are explained
in section 4.1):
</p>
<blockquote>
<pre>
hbool_t H5_libinit_g = FALSE;
</pre>
</blockquote>
<p>
becomes
</p>
<blockquote>
<pre>
H5_api_t H5_g;
</pre>
</blockquote>
<p>
where <code>H5_api_t</code> is
</p>
<blockquote>
<pre>
typedef struct H5_api_struct {
H5_mutex_t init_lock; /* API entrance mutex */
hbool_t H5_libinit_g;
} H5_api_t;
</pre>
</blockquote>
<p>
All former references to <code>H5_libinit_g</code> in the library are now
made using the macro <code>H5_INIT_GLOBAL</code>. If the threadsafe
library is to be used, the macro is set to <code>H5_g.H5_libinit_g</code>
instead.
</p>
<h2>2.2 Global serialization variable</h2>
<p>
A new global boolean variable <code>H5_allow_concurrent_g</code> is used
to determine if multiple threads are allowed to an API call
simultaneously. This is set to <code>FALSE</code>.
</p>
<p>
All APIs that are allowed to do so have their own local variable that
shadows the global variable and is set to <code>TRUE</code>. In phase 1,
no such APIs exist.
</p>
<p>
It is defined in <code>H5.c</code> as follows:
</p>
<blockquote>
<pre>
hbool_t H5_allow_concurrent_g = FALSE;
</pre>
</blockquote>
<h2>2.3 Global thread initialization variable</h2>
<p>
The global variable <code>H5_first_init_g</code> of type
<code>pthread_once_t</code> is used to allow only the first thread in the
application process to call an initialization function using
<code>pthread_once</code>. All subsequent calls to
<code>pthread_once</code> by any thread are disregarded.
</p>
<p>
The call sets up the mutex in the global structure <code>H5_g</code> (see
section 3.1) via an initialization function
<code>H5_first_thread_init</code>. The first thread initialization
function is described in section 4.2.
</p>
<p>
<code>H5_first_init_g</code> is defined in <code>H5.c</code> as follows:
</p>
<blockquote>
<pre>
pthread_once_t H5_first_init_g = PTHREAD_ONCE_INIT;
</pre>
</blockquote>
<h2>2.4 Global key for per-thread error stacks</h2>
<p>
A global pthread-managed key <code>H5_errstk_key_g</code> is used to
allow pthreads to maintain a separate error stack (of type
<code>H5E_t</code>) for each thread. This is defined in <code>H5.c</code>
as:
</p>
<blockquote>
<pre>
pthread_key_t H5_errstk_key_g;
</pre>
</blockquote>
<p>
Error stack management is described in section 4.3.
</p>
<h2>2.5 Global structure and key for thread cancellation prevention</h2>
<p>
We need to preserve the thread cancellation status of each thread
individually by using a key <code>H5_cancel_key_g</code>. The status is
preserved using a structure (of type <code>H5_cancel_t</code>) which
maintains the cancellability state of the thread before it entered the
library and a count (which works very much like the recursive lock
counter) which keeps track of the number of API calls the thread makes
within the library.
</p>
<p>
The structure is defined in <code>H5private.h</code> as:
</p>
<blockquote>
<pre>
/* cancelability structure */
typedef struct H5_cancel_struct {
int previous_state;
unsigned int cancel_count;
} H5_cancel_t;
</pre>
</blockquote>
<p>
Thread cancellation is described in section 4.4.
</p>
<h1>3. Changes to Macro expansions</h1>
<h2>3.1 Changes to FUNC_ENTER</h2>
<p>
The <code>FUNC_ENTER</code> macro is now extended to include macro calls
to initialize first threads, disable cancellability and wraps a lock
operation around the checking of the global initialization flag. It
should be noted that the cancellability should be disabled before
acquiring the lock on the library. Doing so otherwise would allow the
possibility that the thread be cancelled just after it has acquired the
lock on the library and in that scenario, if the cleanup routines are not
properly set, the library would be permanently locked out.
</p>
<p>
The additional macro code and new macro definitions can be found in
Appendix E.1 to E.5. The changes are made in <code>H5private.h</code>.
</p>
<h2>3.2 Changes to HRETURN and HRETURN_ERROR</h2>
<p>
The <code>HRETURN</code> and <code>HRETURN_ERROR</code> macros are the
counterparts to the <code>FUNC_ENTER</code> macro described in section
3.1. <code>FUNC_LEAVE</code> makes a macro call to <code>HRETURN</code>,
so it is also covered here.
</p>
<p>
The basic changes to these two macros involve adding macro calls to call
an unlock operation and re-enable cancellability if necessary. It should
be noted that the cancellability should be re-enabled only after the
thread has released the lock to the library. The consequence of doing
otherwise would be similar to that described in section 3.1.
</p>
<p>
The additional macro code and new macro definitions can be found in
Appendix E.9 to E.9. The changes are made in <code>H5Eprivate.h</code>.
</p>
<h1>4. Implementation of threadsafe functionality</h1>
<h2>4.1 Recursive Locks</h2>
<p>
A recursive mutex lock m allows a thread t1 to successfully lock m more
than once without blocking t1. Another thread t2 will block if t2 tries
to lock m while t1 holds the lock to m. If t1 makes k lock calls on m,
then it also needs to make k unlock calls on m before it releases the
lock.
</p>
<p>
Our implementation of recursive locks is built on top of a pthread mutex
lock (which is not recursive). It makes use of a pthread condition
variable to have unsuccessful threads wait on the mutex. Waiting threads
are awaken by a signal from the final unlock call made by the thread
holding the lock.
</p>
<p>
Recursive locks are defined to be the following type
(<code>H5private.h</code>):
</p>
<blockquote>
<pre>
typedef struct H5_mutex_struct {
pthread_t owner_thread; /* current lock owner */
pthread_mutex_t atomic_lock; /* lock for atomicity of new mechanism */
pthread_cond_t cond_var; /* condition variable */
unsigned int lock_count;
} H5_mutex_t;
</pre>
</blockquote>
<p>
Detailed implementation code can be found in Appendix A. The
implementation changes are made in <code>H5TS.c</code>.
</p>
<h2>4.2 First thread initialization</h2>
<p>
Because the mutex lock associated with a recursive lock cannot be
statically initialized, a mechanism is required to initialize the
recursive lock associated with <code>H5_g</code> so that it can be used
for the first time.
</p>
<p>
The pthreads library allows this through the pthread_once call which as
described in section 3.3 allows only the first thread accessing the
library in an application to initialize <code>H5_g</code>.
</p>
<p>
In addition to initializing <code>H5_g</code>, it also initializes the
key (see section 3.4) for use with per-thread error stacks (see section
4.3).
</p>
<p>
The first thread initialization mechanism is implemented as the function
call <code>H5_first_thread_init()</code> in <code>H5TS.c</code>. This is
described in appendix B.
</p>
<h2>4.3 Per-thread error stack management</h2>
<p>
Pthreads allows individual threads to access dynamic and persistent
per-thread data through the use of keys. Each key is associated with
a table that maps threads to data items. Keys can be initialized by
<code>pthread_key_create()</code> in pthreads (see sections 3.4 and 4.2).
Per-thread data items are accessed using a key through the
<code>pthread_getspecific()</code> and <code>pthread_setspecific()</code>
calls to read and write to the association table respectively.
</p>
<p>
Per-thread error stacks are accessed through the key
<code>H5_errstk_key_g</code> which is initialized by the first thread
initialization call (see section 4.2).
</p>
<p>
In the non-threadsafe version of the library, there is a global stack
variable <code>H5E_stack_g[1]</code> which is no longer defined in the
threadsafe version. At the same time, the macro call to gain access to
the error stack <code>H5E_get_my_stack</code> is changed from:
</p>
<blockquote>
<pre>
#define H5E_get_my_stack() (H5E_stack_g+0)
</pre>
</blockquote>
<p>
to:
</p>
<blockquote>
<pre>
#define H5E_get_my_stack() H5E_get_stack()
</pre>
</blockquote>
<p>
where <code>H5E_get_stack()</code> is a surrogate function that does the
following operations:
</p>
<ol>
<li>if a thread is attempting to get an error stack for the first
time, the error stack is dynamically allocated for the thread and
associated with <code>H5_errstk_key_g</code> using
<code>pthread_setspecific()</code>. The way we detect if it is the
first time is through <code>pthread_getspecific()</code> which
returns <code>NULL</code> if no previous value is associated with
the thread using the key.</li>
<li>if <code>pthread_getspecific()</code> returns a non-null value,
then that is the pointer to the error stack associated with the
thread and the stack can be used as usual.</li>
</ol>
<p>
A final change to the error reporting routines is as follows; the current
implementation reports errors to always be detected at thread 0. In the
threadsafe implementation, this is changed to report the number returned
by a call to <code>pthread_self()</code>.
</p>
<p>
The change in code (reflected in <code>H5Eprint</code> of file
<code>H5E.c</code>) is as follows:
</p>
<blockquote>
<pre>
#ifdef H5_HAVE_THREADSAFE
fprintf (stream, "HDF5-DIAG: Error detected in thread %d."
,pthread_self());
#else
fprintf (stream, "HDF5-DIAG: Error detected in thread 0.");
#endif
</pre>
</blockquote>
<p>
Code for <code>H5E_get_stack()</code> can be found in Appendix C. All the
above changes were made in <code>H5E.c</code>.
</p>
<h2>4.4 Thread Cancellation safety</h2>
<p>
To prevent thread cancellations from killing a thread while it is in the
library, we maintain per-thread information about the cancellability
status of the thread before it entered the library so that we can restore
that same status when the thread leaves the library.
</p>
<p>
By <i>enter</i> and <i>leave</i> the library, we mean the points when a
thread makes an API call from a user application and the time that API
call returns. Other API or callback function calls made from within that
API call are considered <i>within</i> the library.
</p>
<p>
Because other API calls may be made from within the first API call, we
need to maintain a counter to determine which was the first and
correspondingly the last return.
</p>
<p>
When a thread makes an API call, the macro <code>H5_API_SET_CANCEL</code>
calls the worker function <code>H5_cancel_count_inc()</code> which does
the following:
</p>
<ol>
<li>if this is the first time the thread has entered the library,
a new cancellability structure needs to be assigned to it.</li>
<li>if the thread is already within the library when the API call is
made, then cancel_count is simply incremented. Otherwise, we set
the cancellability state to <code>PTHREAD_CANCEL_DISABLE</code>
while storing the previous state into the cancellability structure.
<code>cancel_count</code> is also incremented in this case.</li>
</ol>
<p>
When a thread leaves an API call, the macro
<code>H5_API_UNSET_CANCEL</code> calls the worker function
<code>H5_cancel_count_dec()</code> which does the following:
</p>
<ol>
<li>if <code>cancel_count</code> is greater than 1, indicating that the
thread is not yet about to leave the library, then
<code>cancel_count</code> is simply decremented.</li>
<li>otherwise, we reset the cancellability state back to its original
state before it entered the library and decrement the count (back
to zero).</li>
</ol>
<p>
<code>H5_cancel_count_inc</code> and <code>H5_cancel_count_dec</code> are
described in Appendix D and may be found in <code>H5TS.c</code>.
</p>
<h1>5. Test programs</h1>
<p>
Except where stated, all tests involve 16 simultaneous threads that make
use of HDF-5 API calls without any explicit synchronization typically
required in a non-threadsafe environment.
</p>
<h2>5.1 Data set create and write</h2>
<p>
The test program sets up 16 threads to simultaneously create 16
different datasets named from <i>zero</i> to <i>fifteen</i> for a single
file and then writing an integer value into that dataset equal to the
dataset's named value.
</p>
<p>
The main thread would join with all 16 threads and attempt to match the
resulting HDF-5 file with expected results - that each dataset contains
the correct value (0 for <i>zero</i>, 1 for <i>one</i> etc ...) and all
datasets were correctly created.
</p>
<p>
The test is implemented in the file <code>ttsafe_dcreate.c</code>.
</p>
<h2>5.2 Test on error stack</h2>
<p>
The error stack test is one in which 16 threads simultaneously try to
create datasets with the same name. The result, when properly serialized,
should be equivalent to 16 attempts to create the dataset with the same
name.
</p>
<p>
The error stack implementation runs correctly if it reports 15 instances
of the dataset name conflict error and finally generates a correct HDF-5
containing that single dataset. Each thread should report its own stack
of errors with a thread number associated with it.
</p>
<p>
The test is implemented in the file <code>ttsafe_error.c</code>.
</p>
<h2>5.3 Test on cancellation safety</h2>
<p>
The main idea in thread cancellation safety is as follows; a child thread
is spawned to create and write to a dataset. Following that, it makes a
<code>H5Diterate</code> call on that dataset which activates a callback
function.
</p>
<p>
A deliberate barrier is invoked at the callback function which waits for
both the main and child thread to arrive at that point. After that
happens, the main thread proceeds to make a thread cancel call on the
child thread while the latter sleeps for 3 seconds before proceeding to
write a new value to the dataset.
</p>
<p>
After the iterate call, the child thread logically proceeds to wait
another 3 seconds before writing another newer value to the dataset.
</p>
<p>
The test is correct if the main thread manages to read the second value
at the end of the test. This means that cancellation did not take place
until the end of the iteration call despite of the 3 second wait within
the iteration callback and the extra dataset write operation.
Furthermore, the cancellation should occur before the child can proceed
to write the last value into the dataset.
</p>
<h2>5.4 Test on attribute creation</h2>
<p>
A main thread makes 16 threaded calls to <code>H5Acreate</code> with a
generated name for each attribute. Sixteen attributes should be created
for the single dataset in random (chronological) order and receive values
depending on its generated attribute name (e.g. <i>attrib010</i> would
receive the value 10).
</p>
<p>
After joining with all child threads, the main thread proceeds to read
each attribute by generated name to see if the value tallies. Failure is
detected if the attribute name does not exist (meaning they were never
created) or if the wrong values were read back.
</p>
<h1>A. Recursive Lock implementation code</h1>
<blockquote>
<pre>
void H5_mutex_init(H5_mutex_t *H5_mutex)
{
H5_mutex-&gt;owner_thread = NULL;
pthread_mutex_init(&amp;H5_mutex-&gt;atomic_lock, NULL);
pthread_cond_init(&amp;H5_mutex-&gt;cond_var, NULL);
H5_mutex-&gt;lock_count = 0;
}
void H5_mutex_lock(H5_mutex_t *H5_mutex)
{
pthread_mutex_lock(&amp;H5_mutex-&gt;atomic_lock);
if (pthread_equal(pthread_self(), H5_mutex-&gt;owner_thread)) {
/* already owned by self - increment count */
H5_mutex-&gt;lock_count++;
} else {
if (H5_mutex-&gt;owner_thread == NULL) {
/* no one else has locked it - set owner and grab lock */
H5_mutex-&gt;owner_thread = pthread_self();
H5_mutex-&gt;lock_count = 1;
} else {
/* if already locked by someone else */
while (1) {
pthread_cond_wait(&amp;H5_mutex-&gt;cond_var, &amp;H5_mutex-&gt;atomic_lock);
if (H5_mutex-&gt;owner_thread == NULL) {
H5_mutex-&gt;owner_thread = pthread_self();
H5_mutex-&gt;lock_count = 1;
break;
} /* else do nothing and loop back to wait on condition*/
}
}
}
pthread_mutex_unlock(&amp;H5_mutex-&gt;atomic_lock);
}
void H5_mutex_unlock(H5_mutex_t *H5_mutex)
{
pthread_mutex_lock(&amp;H5_mutex-&gt;atomic_lock);
H5_mutex-&gt;lock_count--;
if (H5_mutex-&gt;lock_count == 0) {
H5_mutex-&gt;owner_thread = NULL;
pthread_cond_signal(&amp;H5_mutex-&gt;cond_var);
}
pthread_mutex_unlock(&amp;H5_mutex-&gt;atomic_lock);
}
</pre>
</blockquote>
<h1>B. First thread initialization</h1>
<blockquote>
<pre>
void H5_first_thread_init(void)
{
/* initialize global API mutex lock */
H5_g.H5_libinit_g = FALSE;
H5_g.init_lock.owner_thread = NULL;
pthread_mutex_init(&amp;H5_g.init_lock.atomic_lock, NULL);
pthread_cond_init(&amp;H5_g.init_lock.cond_var, NULL);
H5_g.init_lock.lock_count = 0;
/* initialize key for thread-specific error stacks */
pthread_key_create(&amp;H5_errstk_key_g, NULL);
/* initialize key for thread cancellability mechanism */
pthread_key_create(&amp;H5_cancel_key_g, NULL);
}
</pre>
</blockquote>
<h1>C. Per-thread error stack acquisition</h1>
<blockquote>
<pre>
H5E_t *H5E_get_stack(void)
{
H5E_t *estack;
if (estack = pthread_getspecific(H5_errstk_key_g)) {
return estack;
} else {
/* no associated value with current thread - create one */
estack = (H5E_t *)malloc(sizeof(H5E_t));
pthread_setspecific(H5_errstk_key_g, (void *)estack);
return estack;
}
}
</pre>
</blockquote>
<h1>D. Thread cancellation mechanisms</h1>
<blockquote>
<pre>
void H5_cancel_count_inc(void)
{
H5_cancel_t *cancel_counter;
if (cancel_counter = pthread_getspecific(H5_cancel_key_g)) {
/* do nothing here */
} else {
/*
* first time thread calls library - create new counter and
* associate with key
*/
cancel_counter = (H5_cancel_t *)malloc(sizeof(H5_cancel_t));
cancel_counter-&gt;cancel_count = 0;
pthread_setspecific(H5_cancel_key_g, (void *)cancel_counter);
}
if (cancel_counter-&gt;cancel_count == 0) {
/* thread entering library */
pthread_setcancelstate(PTHREAD_CANCEL_DISABLE,
&amp;(cancel_counter-&gt;previous_state));
}
cancel_counter-&gt;cancel_count++;
}
void H5_cancel_count_dec(void)
{
H5_cancel_t *cancel_counter = pthread_getspecific(H5_cancel_key_g);
if (cancel_counter-&gt;cancel_count == 1)
pthread_setcancelstate(cancel_counter-&gt;previous_state, NULL);
cancel_counter-&gt;cancel_count--;
}
</pre>
</blockquote>
<h1>E. Macro expansion codes</h1>
<h2>E.1 <code>FUNC_ENTER</code></h2>
<blockquote>
<pre>
/* Initialize the library */ \
H5_FIRST_THREAD_INIT \
H5_API_UNSET_CANCEL \
H5_API_LOCK_BEGIN \
if (!(H5_INIT_GLOBAL)) { \
H5_INIT_GLOBAL = TRUE; \
if (H5_init_library() &lt; 0) { \
HRETURN_ERROR (H5E_FUNC, H5E_CANTINIT, err, \
"library initialization failed"); \
} \
} \
H5_API_LOCK_END \
:
:
:
</pre>
</blockquote>
<h2>E.2 <code>H5_FIRST_THREAD_INIT</code></h2>
<blockquote>
<pre>
/* Macro for first thread initialization */
#define H5_FIRST_THREAD_INIT \
pthread_once(&amp;H5_first_init_g, H5_first_thread_init);
</pre>
</blockquote>
<h2>E.3 <code>H5_API_UNSET_CANCEL</code></h2>
<blockquote>
<pre>
#define H5_API_UNSET_CANCEL \
if (H5_IS_API(FUNC)) { \
H5_cancel_count_inc(); \
}
</pre>
</blockquote>
<h2>E.4 <code>H5_API_LOCK_BEGIN</code></h2>
<blockquote>
<pre>
#define H5_API_LOCK_BEGIN \
if (H5_IS_API(FUNC)) { \
H5_mutex_lock(&amp;H5_g.init_lock);
</pre>
</blockquote>
<h2>E.5 <code>H5_API_LOCK_END</code></h2>
<blockquote>
<pre>
#define H5_API_LOCK_END }
</pre>
</blockquote>
<h2>E.6 <code>HRETURN</code> and <code>HRETURN_ERROR</code></h2>
<blockquote>
<pre>
:
:
H5_API_UNLOCK_BEGIN \
H5_API_UNLOCK_END \
H5_API_SET_CANCEL \
return ret_val; \
}
</pre>
</blockquote>
<h2>E.7 <code>H5_API_UNLOCK_BEGIN</code></h2>
<blockquote>
<pre>
#define H5_API_UNLOCK_BEGIN \
if (H5_IS_API(FUNC)) { \
H5_mutex_unlock(&amp;H5_g.init_lock);
</pre>
</blockquote>
<h2>E.8 <code>H5_API_UNLOCK_END</code></h2>
<blockquote>
<pre>
#define H5_API_UNLOCK_END }
</pre>
</blockquote>
<h2>E.9 <code>H5_API_SET_CANCEL</code></h2>
<blockquote>
<pre>
#define H5_API_SET_CANCEL \
if (H5_IS_API(FUNC)) { \
H5_cancel_count_dec(); \
}
</pre>
</blockquote>
<h2>By Chee Wai Lee</h2>
<h4>By Bill Wendling</h4>
</body>
</html>
doxygen/examples/VFL.html
+1601
View File
@@ -0,0 +1,1601 @@
<HTML>
<HEAD>
<!-- This HTML file has been created by texi2html 1.51
from VFL.texi on 18 November 1999 -->
<TITLE>HDF5 Virtual File Layer</TITLE>
</HEAD>
<!--
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* Copyright by The HDF Group. *
* Copyright by the Board of Trustees of the University of Illinois. *
* All rights reserved. *
* *
* This file is part of HDF5. The full HDF5 copyright notice, including *
* terms governing use, modification, and redistribution, is contained in *
* the files COPYING and Copyright.html. COPYING can be found at the root *
* of the source code distribution tree; Copyright.html can be found at the *
* root level of an installed copy of the electronic HDF5 document set and *
* is linked from the top-level documents page. It can also be found at *
* http://hdfgroup.org/HDF5/doc/Copyright.html. If you do not have *
* access to either file, you may request a copy from help@hdfgroup.org. *
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
-->
<BODY>
<strong>Revision History</strong>
<p>Initial document, 18 November 1999.</p>
<p>Updated on 10/24/00, Quincey Koziol</p>
<p>Added the section &ldquo;Programming Note for C++ Developers Using C
Functions,&rdquo; 08/23/2012, Mark Evans
<P>
<P><HR><P>
<H1>Table of Contents</H1>
<UL>
<LI><A NAME="TOC1" HREF="#SEC1">Introduction</A>
<LI><A NAME="TOC2" HREF="#SEC2">Using a File Driver</A>
<UL>
<LI><A NAME="TOC3" HREF="#SEC3">Driver Header Files</A>
<LI><A NAME="TOC4" HREF="#SEC4">Creating and Opening Files</A>
<LI><A NAME="TOC5" HREF="#SEC5">Performing I/O</A>
<LI><A NAME="TOC6" HREF="#SEC6">File Driver Interchangeability</A>
</UL>
<LI><A NAME="TOC7" HREF="#SEC7">Implementation of a Driver</A>
<UL>
<LI><A NAME="TOC8" HREF="#SEC8">Mode Functions</A>
<LI><A NAME="TOC9" HREF="#SEC9">File Functions</A>
<UL>
<LI><A NAME="TOC10" HREF="#SEC10">Opening Files</A>
<LI><A NAME="TOC11" HREF="#SEC11">Closing Files</A>
<LI><A NAME="TOC12" HREF="#SEC12">File Keys</A>
<LI><A NAME="TOC13" HREF="#SEC13">Saving Modes Across Opens</A>
</UL>
<LI><A NAME="TOC14" HREF="#SEC14">Address Space Functions</A>
<UL>
<LI><A NAME="TOC15" HREF="#SEC15">Userblock and Superblock</A>
<LI><A NAME="TOC16" HREF="#SEC16">Allocation of Format Regions</A>
<LI><A NAME="TOC17" HREF="#SEC17">Freeing Format Regions</A>
<LI><A NAME="TOC18" HREF="#SEC18">Querying Address Range</A>
</UL>
<LI><A NAME="TOC19" HREF="#SEC19">Data Functions</A>
<UL>
<LI><A NAME="TOC20" HREF="#SEC20">Contiguous I/O Functions</A>
<LI><A NAME="TOC21" HREF="#SEC21">Flushing Cached Data</A>
</UL>
<LI><A NAME="TOC22" HREF="#SEC22">Optimization Functions</A>
<LI><A NAME="TOC23" HREF="#SEC23">Registration of a Driver</A>
<ul>
<li><a name="TOCProgNote" href="#SECProgNote">
Programming Note for C++ Developers Using C Functions</a>
</li>
</ul>
<LI><A NAME="TOC24" HREF="#SEC24">Querying Driver Information</A>
</UL>
<LI><A NAME="TOC25" HREF="#SEC25">Miscellaneous</A>
</UL>
<P><HR><P>
<H1><A NAME="SEC1" HREF="#TOC1">Introduction</A></H1>
<P>
The HDF5 file format describes how HDF5 data structures and dataset raw
data are mapped to a linear <STRONG>format address space</STRONG> and the HDF5
library implements that bidirectional mapping in terms of an
API. However, the HDF5 format specifications do <EM>not</EM> indicate how
the format address space is mapped onto storage and HDF (version 5 and
earlier) simply mapped the format address space directly onto a single
file by convention.
</P>
<P>
Since early versions of HDF5 it became apparent that users want the ability to
map the format address space onto different types of storage (a single file,
multiple files, local memory, global memory, network distributed global
memory, a network protocol, <I>etc</I>.) with various types of maps. For
instance, some users want to be able to handle very large format address
spaces on operating systems that support only 2GB files by partitioning the
format address space into equal-sized parts each served by a separate
file. Other users want the same multi-file storage capability but want to
partition the address space according to purpose (raw data in one file, object
headers in another, global heap in a third, <I>etc.</I>) in order to improve I/O
speeds.
</P>
<P>
In fact, the number of storage variations is probably larger than the
number of methods that the HDF5 team is capable of implementing and
supporting. Therefore, a <STRONG>Virtual File Layer</STRONG> API is being
implemented which will allow application teams or departments to design
and implement their own mapping between the HDF5 format address space
and storage, with each mapping being a separate <STRONG>file driver</STRONG>
(possibly written in terms of other file drivers). The HDF5 team will
provide a small set of useful file drivers which will also serve as
examples for those who which to write their own:
</P>
<DL COMPACT>
<DT><CODE>H5FD_SEC2</CODE>
<DD>
This is the default driver which uses Posix file-system functions like
<CODE>read</CODE> and <CODE>write</CODE> to perform I/O to a single file. All I/O
requests are unbuffered although the driver does optimize file seeking
operations to some extent.
<DT><CODE>H5FD_STDIO</CODE>
<DD>
This driver uses functions from <TT>`stdio.h'</TT> to perform buffered I/O
to a single file.
<DT><CODE>H5FD_CORE</CODE>
<DD>
This driver performs I/O directly to memory and can be used to create small
temporary files that never exist on permanent storage. This type of storage is
generally very fast since the I/O consists only of memory-to-memory copy
operations.
<DT><CODE>H5FD_MPIIO</CODE>
<DD>
This is the driver of choice for accessing files in parallel using MPI and
MPI-IO. It is only predefined if the library is compiled with parallel I/O
support.
<DT><CODE>H5FD_FAMILY</CODE>
<DD>
Large format address spaces are partitioned into more manageable pieces and
sent to separate storage locations using an underlying driver of the user's
choice. The <CODE>h5repart</CODE> tool can be used to change the sizes of the
family members when stored as files or to convert a family of files to a
single file or vice versa.
<DT><CODE>H5FD_SPLIT</CODE>
<DD>
The format address space is split into meta data and raw data and each is
mapped onto separate storage using underlying drivers of the user's
choice. The meta data storage can be read by itself (for limited
functionality) or both files can be accessed together.
</DL>
<H1><A NAME="SEC2" HREF="#TOC2">Using a File Driver</A></H1>
<P>
Most application writers will use a driver defined by the HDF5 library or
contributed by another programming team. This chapter describes how existing
drivers are used.
</P>
<H2><A NAME="SEC3" HREF="#TOC3">Driver Header Files</A></H2>
<P>
Each file driver is defined in its own public header file which should
be included by any application which plans to use that driver. The
predefined drivers are in header files whose names begin with
<SAMP>`H5FD'</SAMP> followed by the driver name and <SAMP>`.h'</SAMP>. The <TT>`hdf5.h'</TT>
header file includes all the predefined driver header files.
</P>
<P>
Once the appropriate header file is included a symbol of the form
<SAMP>`H5FD_'</SAMP> followed by the upper-case driver name will be the driver
identification number.<A NAME="DOCF1" HREF="#FOOT1">(1)</A> However, the
value may change if the library is closed (<I>e.g.</I>, by calling
<CODE>H5close</CODE>) and the symbol is referenced again.
</P>
<H2><A NAME="SEC4" HREF="#TOC4">Creating and Opening Files</A></H2>
<P>
In order to create or open a file one must define the method by which the
storage is accessed<A NAME="DOCF2" HREF="#FOOT2">(2)</A> and does so by creating a file access property list<A NAME="DOCF3" HREF="#FOOT3">(3)</A> which is passed to the <CODE>H5Fcreate</CODE> or
<CODE>H5Fopen</CODE> function. A default file access property list is created by
calling <CODE>H5Pcreate</CODE> and then the file driver information is inserted by
calling a driver initialization function such as <CODE>H5Pset_fapl_family</CODE>:
</P>
<PRE>
hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
size_t member_size = 100*1024*1024; /*100MB*/
H5Pset_fapl_family(fapl, member_size, H5P_DEFAULT);
hid_t file = H5Fcreate("foo%05d.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
H5Pclose(fapl);
</PRE>
<P>
Each file driver will have its own initialization function
whose name is <CODE>H5Pset_fapl_</CODE> followed by the driver name and which
takes a file access property list as the first argument followed by
additional driver-dependent arguments.
</P>
<P>
An alternative to using the driver initialization function is to set the
driver directly using the <CODE>H5Pset_driver</CODE> function.<A NAME="DOCF4" HREF="#FOOT4">(4)</A> Its second argument is the file driver identifier, which may
have a different numeric value from run to run depending on the order in which
the file drivers are registered with the library. The third argument
encapsulates the additional arguments of the driver initialization
function. This method only works if the file driver writer has made the
driver-specific property list structure a public datatype, which is
often not the case.
</P>
<PRE>
hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
static H5FD_family_fapl_t fa = {100*1024*1024, H5P_DEFAULT};
H5Pset_driver(fapl, H5FD_FAMILY, &#38;fa);
hid_t file = H5Fcreate("foo.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
H5Pclose(fapl);
</PRE>
<P>
It is also possible to query the file driver information from a file access
property list by calling <CODE>H5Pget_driver</CODE> to determine the driver and then
calling a driver-defined query function to obtain the driver information:
</P>
<PRE>
hid_t driver = H5Pget_driver(fapl);
if (H5FD_SEC2==driver) {
/*nothing further to get*/
} else if (H5FD_FAMILY==driver) {
hid_t member_fapl;
haddr_t member_size;
H5Pget_fapl_family(fapl, &#38;member_size, &#38;member_fapl);
} else if (....) {
....
}
</PRE>
<H2><A NAME="SEC5" HREF="#TOC5">Performing I/O</A></H2>
<P>
The <CODE>H5Dread</CODE> and <CODE>H5Dwrite</CODE> functions transfer data between
application memory and the file. They both take an optional data transfer
property list which has some general driver-independent properties and
optional driver-defined properties. An application will typically perform I/O
in one of three styles via the <CODE>H5Dread</CODE> or <CODE>H5Dwrite</CODE> function:
</P>
<P>
Like file access properties in the previous section, data transfer properties
can be set using a driver initialization function or a general purpose
function. For example, to set the MPI-IO driver to use independent access for
I/O operations one would say:
</P>
<PRE>
hid_t dxpl = H5Pcreate(H5P_DATA_XFER);
H5Pset_dxpl_mpio(dxpl, H5FD_MPIO_INDEPENDENT);
H5Dread(dataset, type, mspace, fspace, buffer, dxpl);
H5Pclose(dxpl);
</PRE>
<P>
The alternative is to initialize a driver defined C <CODE>struct</CODE> and pass it
to the <CODE>H5Pset_driver</CODE> function:
</P>
<PRE>
hid_t dxpl = H5Pcreate(H5P_DATA_XFER);
static H5FD_mpio_dxpl_t dx = {H5FD_MPIO_INDEPENDENT};
H5Pset_driver(dxpl, H5FD_MPIO, &#38;dx);
H5Dread(dataset, type, mspace, fspace, buffer, dxpl);
</PRE>
<P>
The transfer propery list can be queried in a manner similar to the file
access property list: the driver provides a function (or functions) to return
various information about the transfer property list:
</P>
<PRE>
hid_t driver = H5Pget_driver(dxpl);
if (H5FD_MPIO==driver) {
H5FD_mpio_xfer_t xfer_mode;
H5Pget_dxpl_mpio(dxpl, &#38;xfer_mode);
} else {
....
}
</PRE>
<H2><A NAME="SEC6" HREF="#TOC6">File Driver Interchangeability</A></H2>
<P>
The HDF5 specifications describe two things: the mapping of data onto a linear
<STRONG>format address space</STRONG> and the C API which performs the mapping.
However, the mapping of the format address space onto storage intentionally
falls outside the scope of the HDF5 specs. This is a direct result of the fact
that it is not generally possible to store information about how to access
storage inside the storage itself. For instance, given only the file name
<TT>`/arborea/1225/work/f%03d'</TT> the HDF5 library is unable to tell whether the
name refers to a file on the local file system, a family of files on the local
file system, a file on host <SAMP>`arborea'</SAMP> port 1225, a family of files on a
remote system, <I>etc</I>.
</P>
<P>
Two ways which library could figure out where the storage is located are:
storage access information can be provided by the user, or the library can try
all known file access methods. This implementation uses the former method.
</P>
<P>
In general, if a file was created with one driver then it isn't possible to
open it with another driver. There are of course exceptions: a file created
with MPIO could probably be opened with the sec2 driver, any file created
by the sec2 driver could be opened as a family of files with one member,
<I>etc</I>. In fact, sometimes a file must not only be opened with the same
driver but also with the same driver properties. The predefined drivers are
written in such a way that specifying the correct driver is sufficient for
opening a file.
</P>
<H1><A NAME="SEC7" HREF="#TOC7">Implementation of a Driver</A></H1>
<P>
A driver is simply a collection of functions and data structures which are
registered with the HDF5 library at runtime. The functions fall into these
categories:
</P>
<UL>
<LI>Functions which operate on modes
<LI>Functions which operate on files
<LI>Functions which operate on the address space
<LI>Functions which operate on data
<LI>Functions for driver initialization
<LI>Optimization functions
</UL>
<H2><A NAME="SEC8" HREF="#TOC8">Mode Functions</A></H2>
<P>
Some drivers need information about file access and data transfers which are
very specific to the driver. The information is usually implemented as a pair
of pointers to C structs which are allocated and initialized as part of an
HDF5 property list and passed down to various driver functions. There are two
classes of settings: file access modes that describe how to access the file
through the driver, and data transfer modes which are settings that control
I/O operations. Each file opened by a particular driver may have a different
access mode; each dataset I/O request for a particular file may have a
different data transfer mode.
</P>
<P>
Since each driver has its own particular requirements for various settings,
each driver is responsible for defining the mode structures that it
needs. Higher layers of the library treat the structures as opaque but must be
able to copy and free them. Thus, the driver provides either the size of the
structure or a pair of function pointers for each of the mode types.
</P>
<P>
<STRONG>Example:</STRONG> The family driver needs to know how the format address
space is partitioned and the file access property list to use for the
family members.
</P>
<PRE>
/* Driver-specific file access properties */
typedef struct H5FD_family_fapl_t {
hsize_t memb_size; /*size of each member */
hid_t memb_fapl_id; /*file access property list of each memb*/
} H5FD_family_fapl_t;
/* Driver specific data transfer properties */
typedef struct H5FD_family_dxpl_t {
hid_t memb_dxpl_id; /*data xfer property list of each memb */
} H5FD_family_dxpl_t;
</PRE>
<P>
In order to copy or free one of these structures the member file access
or data transfer properties must also be copied or freed. This is done
by providing a copy and close function for each structure:
</P>
<P>
<STRONG>Example:</STRONG> The file access property list copy and close functions
for the family driver:
</P>
<PRE>
static void *
H5FD_family_fapl_copy(const void *_old_fa)
{
const H5FD_family_fapl_t *old_fa = (const H5FD_family_fapl_t*)_old_fa;
H5FD_family_fapl_t *new_fa = malloc(sizeof(H5FD_family_fapl_t));
assert(new_fa);
memcpy(new_fa, old_fa, sizeof(H5FD_family_fapl_t));
new_fa-&#62;memb_fapl_id = H5Pcopy(old_fa-&#62;memb_fapl_id);
return new_fa;
}
static herr_t
H5FD_family_fapl_free(void *_fa)
{
H5FD_family_fapl_t *fa = (H5FD_family_fapl_t*)_fa;
H5Pclose(fa-&#62;memb_fapl_id);
free(fa);
return 0;
}
</PRE>
<P>
Generally when a file is created or opened the file access properties
for the driver are copied into the file pointer which is returned and
they may be modified from their original value (for instance, the file
family driver modifies the member size property when opening an existing
family). In order to support the <CODE>H5Fget_access_plist</CODE> function the
driver must provide a <CODE>fapl_get</CODE> callback which creates a copy of
the driver-specific properties based on a particular file.
</P>
<P>
<STRONG>Example:</STRONG> The file family driver copies the member size file
access property list into the return value:
</P>
<PRE>
static void *
H5FD_family_fapl_get(H5FD_t *_file)
{
H5FD_family_t *file = (H5FD_family_t*)_file;
H5FD_family_fapl_t *fa = calloc(1, sizeof(H5FD_family_fapl_t*));
fa-&#62;memb_size = file-&#62;memb_size;
fa-&#62;memb_fapl_id = H5Pcopy(file-&#62;memb_fapl_id);
return fa;
}
</PRE>
<H2><A NAME="SEC9" HREF="#TOC9">File Functions</A></H2>
<P>
The higher layers of the library expect files to have a name and allow the
file to be accessed in various modes. The driver must be able to create a new
file, replace an existing file, or open an existing file. Opening or creating
a file should return a handle, a pointer to a specialization of the
<CODE>H5FD_t</CODE> struct, which allows read-only or read-write access and which
will be passed to the other driver functions as they are
called.<A NAME="DOCF5" HREF="#FOOT5">(5)</A>
</P>
<PRE>
typedef struct {
/* Public fields */
H5FD_class_t *cls; /*class data defined below*/
/* Private fields -- driver-defined */
} H5FD_t;
</PRE>
<P>
<STRONG>Example:</STRONG> The family driver requires handles to the underlying
storage, the size of the members for this particular file (which might be
different than the member size specified in the file access property list if
an existing file family is being opened), the name used to open the file in
case additional members must be created, and the flags to use for creating
those additional members. The <CODE>eoa</CODE> member caches the size of the format
address space so the family members don't have to be queried in order to find
it.
</P>
<PRE>
/* The description of a file belonging to this driver. */
typedef struct H5FD_family_t {
H5FD_t pub; /*public stuff, must be first */
hid_t memb_fapl_id; /*file access property list for members */
hsize_t memb_size; /*maximum size of each member file */
int nmembs; /*number of family members */
int amembs; /*number of member slots allocated */
H5FD_t **memb; /*dynamic array of member pointers */
haddr_t eoa; /*end of allocated addresses */
char *name; /*name generator printf format */
unsigned flags; /*flags for opening additional members */
} H5FD_family_t;
</PRE>
<P>
<STRONG>Example:</STRONG> The sec2 driver needs to keep track of the underlying Unix
file descriptor and also the end of format address space and current Unix file
size. It also keeps track of the current file position and last operation
(read, write, or unknown) in order to optimize calls to <CODE>lseek</CODE>. The
<CODE>device</CODE> and <CODE>inode</CODE> fields are defined on Unix in order to uniquely
identify the file and will be discussed below.
</P>
<PRE>
typedef struct H5FD_sec2_t {
H5FD_t pub; /*public stuff, must be first */
int fd; /*the unix file */
haddr_t eoa; /*end of allocated region */
haddr_t eof; /*end of file; current file size*/
haddr_t pos; /*current file I/O position */
int op; /*last operation */
dev_t device; /*file device number */
ino_t inode; /*file i-node number */
} H5FD_sec2_t;
</PRE>
<H3><A NAME="SEC10" HREF="#TOC10">Opening Files</A></H3>
<P>
All drivers must define a function for opening/creating a file. This
function should have a prototype which is:
</P>
<P>
<DL>
<DT><U>Function:</U> static H5FD_t * <B>open</B> <I>(const char *<VAR>name</VAR>, unsigned <VAR>flags</VAR>, hid_t <VAR>fapl</VAR>, haddr_t <VAR>maxaddr</VAR>)</I>
<DD><A NAME="IDX1"></A>
</P>
<P>
The file name <VAR>name</VAR> and file access property list <VAR>fapl</VAR> are
the same as were specified in the <CODE>H5Fcreate</CODE> or <CODE>H5Fopen</CODE>
call. The <VAR>flags</VAR> are the same as in those calls also except the
flag <CODE>H5F_ACC_CREATE</CODE> is also present if the call was to
<CODE>H5Fcreate</CODE> and they are documented in the <TT>`H5Fpublic.h'</TT>
file. The <VAR>maxaddr</VAR> argument is the maximum format address that the
driver should be prepared to handle (the minimum address is always
zero).
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver opens a Unix file with the requested name
and saves information which uniquely identifies the file (the Unix device
number and inode).
</P>
<PRE>
static H5FD_t *
H5FD_sec2_open(const char *name, unsigned flags, hid_t fapl_id/*unused*/,
haddr_t maxaddr)
{
unsigned o_flags;
int fd;
struct stat sb;
H5FD_sec2_t *file=NULL;
/* Check arguments */
if (!name || !*name) return NULL;
if (0==maxaddr || HADDR_UNDEF==maxaddr) return NULL;
if (ADDR_OVERFLOW(maxaddr)) return NULL;
/* Build the open flags */
o_flags = (H5F_ACC_RDWR &#38; flags) ? O_RDWR : O_RDONLY;
if (H5F_ACC_TRUNC &#38; flags) o_flags |= O_TRUNC;
if (H5F_ACC_CREAT &#38; flags) o_flags |= O_CREAT;
if (H5F_ACC_EXCL &#38; flags) o_flags |= O_EXCL;
/* Open the file */
if ((fd=open(name, o_flags, 0666))&#60;0) return NULL;
if (fstat(fd, &#38;sb)&#60;0) {
close(fd);
return NULL;
}
/* Create the new file struct */
file = calloc(1, sizeof(H5FD_sec2_t));
file-&#62;fd = fd;
file-&#62;eof = sb.st_size;
file-&#62;pos = HADDR_UNDEF;
file-&#62;op = OP_UNKNOWN;
file-&#62;device = sb.st_dev;
file-&#62;inode = sb.st_ino;
return (H5FD_t*)file;
}
</PRE>
<H3><A NAME="SEC11" HREF="#TOC11">Closing Files</A></H3>
<P>
Closing a file simply means that all cached data should be flushed to the next
lower layer, the file should be closed at the next lower layer, and all
file-related data structures should be freed. All information needed by the
close function is already present in the file handle.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>close</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX2"></A>
</P>
<P>
The <VAR>file</VAR> argument is the handle which was returned by the <CODE>open</CODE>
function, and the <CODE>close</CODE> should free only memory associated with the
driver-specific part of the handle (the public parts will have already been released by HDF5's virtual file layer).
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver just closes the underlying Unix file,
making sure that the actual file size is the same as that known to the
library by writing a zero to the last file position it hasn't been
written by some previous operation (which happens in the same code which
flushes the file contents and is shown below).
</P>
<PRE>
static herr_t
H5FD_sec2_close(H5FD_t *_file)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
if (H5FD_sec2_flush(_file)&#60;0) return -1;
if (close(file-&#62;fd)&#60;0) return -1;
free(file);
return 0;
}
</PRE>
<H3><A NAME="SEC12" HREF="#TOC12">File Keys</A></H3>
<P>
Occasionally an application will attempt to open a single file more than one
time in order to obtain multiple handles to the file. HDF5 allows the files to
share information<A NAME="DOCF6" HREF="#FOOT6">(6)</A> but in order to
accomplish this HDF5 must be able to tell when two names refer to the same
file. It does this by associating a driver-defined key with each file opened
by a driver and comparing the key for an open request with the keys for all
other files currently open by the same driver.
</P>
<P>
<DL>
<DT><U>Function:</U> const int <B>cmp</B> <I>(const H5FD_t *<VAR>f1</VAR>, const H5FD_t *<VAR>f2</VAR>)</I>
<DD><A NAME="IDX3"></A>
</P>
<P>
The driver may provide a function which compares two files <VAR>f1</VAR> and
<VAR>f2</VAR> belonging to the same driver and returns a negative, positive, or
zero value <I>a la</I> the <CODE>strcmp</CODE> function.<A NAME="DOCF7" HREF="#FOOT7">(7)</A> If this
function is not provided then HDF5 assumes that all calls to the <CODE>open</CODE>
callback return unique files regardless of the arguments and it is up to the
application to avoid doing this if that assumption is incorrect.
</DL>
</P>
<P>
Each time a file is opened the library calls the <CODE>cmp</CODE> function to
compare that file with all other files currently open by the same driver and
if one of them matches (at most one can match) then the file which was just
opened is closed and the previously opened file is used instead.
</P>
<P>
Opening a file twice with incompatible flags will result in failure. For
instance, opening a file with the truncate flag is a two step process which
first opens the file without truncation so keys can be compared, and if no
matching file is found already open then the file is closed and immediately
reopened with the truncation flag set (if a matching file is already open then
the truncating open will fail).
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver uses the Unix device and i-node as the
key. They were initialized when the file was opened.
</P>
<PRE>
static int
H5FD_sec2_cmp(const H5FD_t *_f1, const H5FD_t *_f2)
{
const H5FD_sec2_t *f1 = (const H5FD_sec2_t*)_f1;
const H5FD_sec2_t *f2 = (const H5FD_sec2_t*)_f2;
if (f1-&#62;device &#60; f2-&#62;device) return -1;
if (f1-&#62;device &#62; f2-&#62;device) return 1;
if (f1-&#62;inode &#60; f2-&#62;inode) return -1;
if (f1-&#62;inode &#62; f2-&#62;inode) return 1;
return 0;
}
</PRE>
<H3><A NAME="SEC13" HREF="#TOC13">Saving Modes Across Opens</A></H3>
<P>
Some drivers may also need to store certain information in the file superblock
in order to be able to reliably open the file at a later date. This is done by
three functions: one to determine how much space will be necessary to store
the information in the superblock, one to encode the information, and one to
decode the information. These functions are optional, but if any one is
defined then the other two must also be defined.
</P>
<P>
<DL>
<DT><U>Function:</U> static hsize_t <B>sb_size</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX4"></A>
<DT><U>Function:</U> static herr_t <B>sb_encode</B> <I>(H5FD_t *<VAR>file</VAR>, char *<VAR>name</VAR>, unsigned char *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX5"></A>
<DT><U>Function:</U> static herr_t <B>sb_decode</B> <I>(H5FD_t *<VAR>file</VAR>, const char *<VAR>name</VAR>, const unsigned char *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX6"></A>
</P>
<P>
The <CODE>sb_size</CODE> function returns the number of bytes necessary to encode
information needed later if the file is reopened. The <CODE>sb_encode</CODE>
function encodes information from the file into buffer <VAR>buf</VAR>
allocated by the caller. It also writes an 8-character (plus null
termination) into the <CODE>name</CODE> argument, which should be a unique
identification for the driver. The <CODE>sb_decode</CODE> function looks at
the <VAR>name</VAR>
</P>
<P>
decodes
data from the buffer <VAR>buf</VAR> and updates the <VAR>file</VAR> argument with the new information,
advancing <VAR>*p</VAR> in the process.
</DL>
</P>
<P>
The part of this which is somewhat tricky is that the file must be readable
before the superblock information is decoded. File access modes fall outside
the scope of the HDF5 file format, but they are placed inside the boot block
for convenience.<A NAME="DOCF8" HREF="#FOOT8">(8)</A>
</P>
<P>
<STRONG>Example:</STRONG> <EM>To be written later.</EM>
</P>
<H2><A NAME="SEC14" HREF="#TOC14">Address Space Functions</A></H2>
<P>
HDF5 does not assume that a file is a linear address space of bytes. Instead,
the library will call functions to allocate and free portions of the HDF5
format address space, which in turn map onto functions in the file driver to
allocate and free portions of file address space. The library tells the file
driver how much format address space it wants to allocate and the driver
decides what format address to use and how that format address is mapped onto
the file address space. Usually the format address is chosen so that the file
address can be calculated in constant time for data I/O operations (which are
always specified by format addresses).
</P>
<H3><A NAME="SEC15" HREF="#TOC15">Userblock and Superblock</A></H3>
<P>
The HDF5 format allows an optional userblock to appear before the actual HDF5
data in such a way that if the userblock is <STRONG>sucked out</STRONG> of the file and
everything remaining is shifted downward in the file address space, then the
file is still a valid HDF5 file. The userblock size can be zero or any
multiple of two greater than or equal to 512 and the file superblock begins
immediately after the userblock.
</P>
<P>
HDF5 allocates space for the userblock and superblock by calling an
allocation function defined below, which must return a chunk of memory at
format address zero on the first call.
</P>
<H3><A NAME="SEC16" HREF="#TOC16">Allocation of Format Regions</A></H3>
<P>
The library makes many types of allocation requests:
</P>
<DL COMPACT>
<DT><CODE>H5FD_MEM_SUPER</CODE>
<DD>
An allocation request for the userblock and/or superblock.
<DT><CODE>H5FD_MEM_BTREE</CODE>
<DD>
An allocation request for a node of a B-tree.
<DT><CODE>H5FD_MEM_DRAW</CODE>
<DD>
An allocation request for the raw data of a dataset.
<DT><CODE>H5FD_MEM_META</CODE>
<DD>
An allocation request for the raw data of a dataset which
the user has indicated will be relatively small.
<DT><CODE>H5FD_MEM_GROUP</CODE>
<DD>
An allocation request for a group leaf node (internal nodes of the group tree
are allocated as H5MF_BTREE).
<DT><CODE>H5FD_MEM_GHEAP</CODE>
<DD>
An allocation request for a global heap collection. Global heaps are used to
store certain types of references such as dataset region references. The set
of all global heap collections can become quite large.
<DT><CODE>H5FD_MEM_LHEAP</CODE>
<DD>
An allocation request for a local heap. Local heaps are used to store the
names which are members of a group. The combined size of all local heaps is a
function of the number of object names in the file.
<DT><CODE>H5FD_MEM_OHDR</CODE>
<DD>
An allocation request for (part of) an object header. Object headers are
relatively small and include meta information about objects (like the data
space and type of a dataset) and attributes.
</DL>
<P>
When a chunk of memory is freed the library adds it to a free list and
allocation requests are satisfied from the free list before requesting memory
from the file driver. Each type of allocation request enumerated above has its
own free list, but the file driver can specify that certain object types can
share a free list. It does so by providing an array which maps a request type
to a free list. If any value of the map is <CODE>H5MF_DEFAULT</CODE> (zero) then the
object's own free list is used. The special value <CODE>H5MF_NOLIST</CODE> indicates
that the library should not attempt to maintain a free list for that
particular object type, instead calling the file driver each time an object of
that type is freed.
</P>
<P>
Mappings predefined in the <TT>`H5FDpublic.h'</TT> file are:
<DL COMPACT>
<DT><CODE>H5FD_FLMAP_SINGLE</CODE>
<DD>
All memory usage types are mapped to a single free list.
<DT><CODE>H5FD_FLMAP_DICHOTOMY</CODE>
<DD>
Memory usage is segregated into meta data and raw data for the purposes of
memory management.
<DT><CODE>H5FD_FLMAP_DEFAULT</CODE>
<DD>
Each memory usage type has its own free list.
</DL>
<P>
<STRONG>Example:</STRONG> To make a map that manages object headers on one free list
and everything else on another free list one might initialize the map with the
following code: (the use of <CODE>H5FD_MEM_SUPER</CODE> is arbitrary)
</P>
<PRE>
H5FD_mem_t mt, map[H5FD_MEM_NTYPES];
for (mt=0; mt&#60;H5FD_MEM_NTYPES; mt++) {
map[mt] = (H5FD_MEM_OHDR==mt) ? mt : H5FD_MEM_SUPER;
}
</PRE>
<P>
If an allocation request cannot be satisfied from the free list then one of
two things happen. If the driver defines an allocation callback then it is
used to allocate space; otherwise new memory is allocated from the end of the
format address space by incrementing the end-of-address marker.
</P>
<P>
<DL>
<DT><U>Function:</U> static haddr_t <B>alloc</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, hsize_t <VAR>size</VAR>)</I>
<DD><A NAME="IDX7"></A>
</P>
<P>
The <VAR>file</VAR> argument is the file from which space is to be allocated,
<VAR>type</VAR> is the type of memory being requested (from the list above) without
being mapped according to the freelist map and <VAR>size</VAR> is the number of
bytes being requested. The library is allowed to allocate large chunks of
storage and manage them in a layer above the file driver (although the current
library doesn't do that). The allocation function should return a format
address for the first byte allocated. The allocated region extends from that
address for <VAR>size</VAR> bytes. If the request cannot be honored then the
undefined address value is returned (<CODE>HADDR_UNDEF</CODE>). The first call to
this function for a file which has never had memory allocated <EM>must</EM>
return a format address of zero or <CODE>HADDR_UNDEF</CODE> since this is how the
library allocates space for the userblock and/or superblock.
</DL>
</P>
<P>
<STRONG>Example:</STRONG> <EM>To be written later.</EM>
</P>
<H3><A NAME="SEC17" HREF="#TOC17">Freeing Format Regions</A></H3>
<P>
When the library is finished using a certain region of the format address
space it will return the space to the free list according to the type of
memory being freed and the free list map described above. If the free list has
been disabled for a particular memory usage type (according to the free list
map) and the driver defines a <CODE>free</CODE> callback then it will be
invoked. The <CODE>free</CODE> callback is also invoked for all entries on the free
list when the file is closed.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>free</B> <I>(H5FD_t *<VAR>file</VAR>, H5MF_type_t <VAR>type</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>)</I>
<DD><A NAME="IDX8"></A>
</P>
<P>
The <VAR>file</VAR> argument is the file for which space is being freed; <VAR>type</VAR>
is the type of object being freed (from the list above) without being mapped
according to the freelist map; <VAR>addr</VAR> is the first format address to free;
and <VAR>size</VAR> is the size in bytes of the region being freed. The region
being freed may refer to just part of the region originally allocated and/or
may cross allocation boundaries provided all regions being freed have the same
usage type. However, the library will never attempt to free regions which have
already been freed or which have never been allocated.
</DL>
</P>
<P>
A driver may choose to not define the <CODE>free</CODE> function, in which case
format addresses will be leaked. This isn't normally a huge problem since the
library contains a simple free list of its own and freeing parts of the format
address space is not a common occurrence.
</P>
<P>
<STRONG>Example:</STRONG> <EM>To be written later.</EM>
</P>
<H3><A NAME="SEC18" HREF="#TOC18">Querying Address Range</A></H3>
<P>
Each file driver must have some mechanism for setting and querying the end of
address, or <STRONG>EOA</STRONG>, marker. The EOA marker is the first format address
after the last format address ever allocated. If the last part of the
allocated address range is freed then the driver may optionally decrease the
eoa marker.
</P>
<P>
<DL>
<DT><U>Function:</U> static haddr_t <B>get_eoa</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX9"></A>
</P>
<P>
This function returns the current value of the EOA marker for the specified
file.
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver just returns the current eoa marker value
which is cached in the file structure:
</P>
<PRE>
static haddr_t
H5FD_sec2_get_eoa(H5FD_t *_file)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
return file-&#62;eoa;
}
</PRE>
<P>
The eoa marker is initially zero when a file is opened and the library may set
it to some other value shortly after the file is opened (after the superblock
is read and the saved eoa marker is determined) or when allocating additional
memory in the absence of an <CODE>alloc</CODE> callback (described above).
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver simply caches the eoa marker in the file
structure and does not extend the underlying Unix file. When the file is
flushed or closed then the Unix file size is extended to match the eoa marker.
</P>
<PRE>
static herr_t
H5FD_sec2_set_eoa(H5FD_t *_file, haddr_t addr)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
file-&#62;eoa = addr;
return 0;
}
</PRE>
<H2><A NAME="SEC19" HREF="#TOC19">Data Functions</A></H2>
<P>
These functions operate on data, transferring a region of the format address
space between memory and files.
</P>
<H3><A NAME="SEC20" HREF="#TOC20">Contiguous I/O Functions</A></H3>
<P>
A driver must specify two functions to transfer data from the library to the
file and vice versa.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>read</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, void *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX10"></A>
<DT><U>Function:</U> static herr_t <B>write</B> <I>(H5FD_t *<VAR>file</VAR>, H5FD_mem_t <VAR>type</VAR>, hid_t <VAR>dxpl</VAR>, haddr_t <VAR>addr</VAR>, hsize_t <VAR>size</VAR>, const void *<VAR>buf</VAR>)</I>
<DD><A NAME="IDX11"></A>
</P>
<P>
The <CODE>read</CODE> function reads data from file <VAR>file</VAR> beginning at address
<VAR>addr</VAR> and continuing for <VAR>size</VAR> bytes into the buffer <VAR>buf</VAR>
supplied by the caller. The <CODE>write</CODE> function transfers data in the
opposite direction. Both functions take a data transfer property list
<VAR>dxpl</VAR> which indicates the fine points of how the data is to be
transferred and which comes directly from the <CODE>H5Dread</CODE> or
<CODE>H5Dwrite</CODE> function. Both functions receive <VAR>type</VAR> of
data being written, which may allow a driver to tune it's behavior for
different kinds of data.
</DL>
</P>
<P>
Both functions should return a negative value if they fail to transfer the
requested data, or non-negative if they succeed. The library will never
attempt to read from unallocated regions of the format address space.
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver just makes system calls. It tries not to
call <CODE>lseek</CODE> if the current operation is the same as the previous
operation and the file position is correct. It also fills the output buffer
with zeros when reading between the current EOF and EOA markers and restarts
system calls which were interrupted.
</P>
<PRE>
static herr_t
H5FD_sec2_read(H5FD_t *_file, H5FD_mem_t type/*unused*/, hid_t dxpl_id/*unused*/,
haddr_t addr, hsize_t size, void *buf/*out*/)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
ssize_t nbytes;
assert(file &#38;&#38; file-&#62;pub.cls);
assert(buf);
/* Check for overflow conditions */
if (REGION_OVERFLOW(addr, size)) return -1;
if (addr+size&#62;file-&#62;eoa) return -1;
/* Seek to the correct location */
if ((addr!=file-&#62;pos || OP_READ!=file-&#62;op) &#38;&#38;
file_seek(file-&#62;fd, (file_offset_t)addr, SEEK_SET)&#60;0) {
file-&#62;pos = HADDR_UNDEF;
file-&#62;op = OP_UNKNOWN;
return -1;
}
/*
* Read data, being careful of interrupted system calls, partial results,
* and the end of the file.
*/
while (size&#62;0) {
do nbytes = read(file-&#62;fd, buf, size);
while (-1==nbytes &#38;&#38; EINTR==errno);
if (-1==nbytes) {
/* error */
file-&#62;pos = HADDR_UNDEF;
file-&#62;op = OP_UNKNOWN;
return -1;
}
if (0==nbytes) {
/* end of file but not end of format address space */
memset(buf, 0, size);
size = 0;
}
assert(nbytes&#62;=0);
assert((hsize_t)nbytes&#60;=size);
size -= (hsize_t)nbytes;
addr += (haddr_t)nbytes;
buf = (char*)buf + nbytes;
}
/* Update current position */
file-&#62;pos = addr;
file-&#62;op = OP_READ;
return 0;
}
</PRE>
<P>
<STRONG>Example:</STRONG> The sec2 <CODE>write</CODE> callback is similar except it updates
the file EOF marker when extending the file.
</P>
<H3><A NAME="SEC21" HREF="#TOC21">Flushing Cached Data</A></H3>
<P>
Some drivers may desire to cache data in memory in order to make larger I/O
requests to the underlying file and thus improving bandwidth. Such drivers
should register a cache flushing function so that the library can insure that
data has been flushed out of the drivers in response to the application
calling <CODE>H5Fflush</CODE>.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>flush</B> <I>(H5FD_t *<VAR>file</VAR>)</I>
<DD><A NAME="IDX12"></A>
</P>
<P>
Flush all data for file <VAR>file</VAR> to storage.
</DL>
</P>
<P>
<STRONG>Example:</STRONG> The sec2 driver doesn't cache any data but it also doesn't
extend the Unix file as agressively as it should. Therefore, when finalizing a
file it should write a zero to the last byte of the allocated region so that
when reopening the file later the EOF marker will be at least as large as the
EOA marker saved in the superblock (otherwise HDF5 will refuse to open the
file, claiming that the data appears to be truncated).
</P>
<PRE>
static herr_t
H5FD_sec2_flush(H5FD_t *_file)
{
H5FD_sec2_t *file = (H5FD_sec2_t*)_file;
if (file-&#62;eoa&#62;file-&#62;eof) {
if (-1==file_seek(file-&#62;fd, file-&#62;eoa-1, SEEK_SET)) return -1;
if (write(file-&#62;fd, "", 1)!=1) return -1;
file-&#62;eof = file-&#62;eoa;
file-&#62;pos = file-&#62;eoa;
file-&#62;op = OP_WRITE;
}
return 0;
}
</PRE>
<H2><A NAME="SEC22" HREF="#TOC22">Optimization Functions</A></H2>
<P>
The library is capable of performing several generic optimizations on I/O, but
these types of optimizations may not be appropriate for a given VFL driver.
</P>
<P>
Each driver may provide a query function to allow the library to query whether
to enable these optimizations. If a driver lacks a query function, the library
will disable all types of optimizations which can be queried.
</P>
<P>
<DL>
<DT><U>Function:</U> static herr_t <B>query</B> <I>(const H5FD_t *<VAR>file</VAR>, unsigned long *<VAR>flags</VAR>)</I>
<DD><A NAME="IDX17"></A>
</P>
<P>
This function is called by the library to query which optimizations to enable
for I/O to this driver. These are the flags which are currently defined:
<UL>
<DL>
<DT>H5FD_FEAT_AGGREGATE_METADATA (0x00000001)
<DD>Defining the H5FD_FEAT_AGGREGATE_METADATA for a VFL driver means that
the library will attempt to allocate a larger block for metadata and
then sub-allocate each metadata request from that larger block.
<DT>H5FD_FEAT_ACCUMULATE_METADATA (0x00000002)
<DD>Defining the H5FD_FEAT_ACCUMULATE_METADATA for a VFL driver means that
the library will attempt to cache metadata as it is written to the file
and build up a larger block of metadata to eventually pass to the VFL
'write' routine.
<DT>H5FD_FEAT_DATA_SIEVE (0x00000004)
<DD>Defining the H5FD_FEAT_DATA_SIEVE for a VFL driver means that
the library will attempt to cache raw data as it is read from/written to
a file in a "data sieve" buffer. See Rajeev Thakur's papers:
<UL>
<DL>
<DT>http://www.mcs.anl.gov/~thakur/papers/romio-coll.ps.gz
<DT>http://www.mcs.anl.gov/~thakur/papers/mpio-high-perf.ps.gz
</DL>
</UL>
</DL>
</UL>
</P>
</DL>
</P>
<H2><A NAME="SEC23" HREF="#TOC23">Registration of a Driver</A></H2>
<P>
Before a driver can be used the HDF5 library needs to be told of its
existence. This is done by registering the driver, which results in a driver
identification number. Instead of passing many arguments to the registration
function, the driver information is entered into a structure and the address
of the structure is passed to the registration function where it is
copied. This allows the HDF5 API to be extended while providing backward
compatibility at the source level.
</P>
<P>
<DL>
<DT><U>Function:</U> hid_t <B>H5FDregister</B> <I>(H5FD_class_t *<VAR>cls</VAR>)</I>
<DD><A NAME="IDX13"></A>
</P>
<P>
The driver described by struct <VAR>cls</VAR> is registered with the library and an
ID number for the driver is returned.
</DL>
</P>
<P>
The <CODE>H5FD_class_t</CODE> type is a struct with the following fields:
</P>
<DL COMPACT>
<DT><CODE>const char *name</CODE>
<DD>
A pointer to a constant, null-terminated driver name to be used for debugging
purposes.
<DT><CODE>size_t fapl_size</CODE>
<DD>
The size in bytes of the file access mode structure or zero if the driver
supplies a copy function or doesn't define the structure.
<DT><CODE>void *(*fapl_copy)(const void *fapl)</CODE>
<DD>
An optional function which copies a driver-defined file access mode structure.
This field takes precedence over <CODE>fm_size</CODE> when both are defined.
<DT><CODE>void (*fapl_free)(void *fapl)</CODE>
<DD>
An optional function to free the driver-defined file access mode structure. If
null, then the library calls the C <CODE>free</CODE> function to free the
structure.
<DT><CODE>size_t dxpl_size</CODE>
<DD>
The size in bytes of the data transfer mode structure or zero if the driver
supplies a copy function or doesn't define the structure.
<DT><CODE>void *(*dxpl_copy)(const void *dxpl)</CODE>
<DD>
An optional function which copies a driver-defined data transfer mode
structure. This field takes precedence over <CODE>xm_size</CODE> when both are
defined.
<DT><CODE>void (*dxpl_free)(void *dxpl)</CODE>
<DD>
An optional function to free the driver-defined data transfer mode
structure. If null, then the library calls the C <CODE>free</CODE> function to
free the structure.
<DT><CODE>H5FD_t *(*open)(const char *name, unsigned flags, hid_t fapl, haddr_t maxaddr)</CODE>
<DD>
The function which opens or creates a new file.
<DT><CODE>herr_t (*close)(H5FD_t *file)</CODE>
<DD>
The function which ends access to a file.
<DT><CODE>int (*cmp)(const H5FD_t *f1, const H5FD_t *f2)</CODE>
<DD>
An optional function to determine whether two open files have the same key. If
this function is not present then the library assumes that two files will
never be the same.
<DT><CODE>int (*query)(const H5FD_t *f, unsigned long *flags)</CODE>
<DD>
An optional function to determine which library optimizations a driver can
support.
<DT><CODE>haddr_t (*alloc)(H5FD_t *file, H5FD_mem_t type, hsize_t size)</CODE>
<DD>
An optional function to allocate space in the file.
<DT><CODE>herr_t (*free)(H5FD_t *file, H5FD_mem_t type, haddr_t addr, hsize_t size)</CODE>
<DD>
An optional function to free space in the file.
<DT><CODE>haddr_t (*get_eoa)(H5FD_t *file)</CODE>
<DD>
A function to query how much of the format address space has been allocated.
<DT><CODE>herr_t (*set_eoa)(H5FD_t *file, haddr_t)</CODE>
<DD>
A function to set the end of address space.
<DT><CODE>haddr_t (*get_eof)(H5FD_t *file)</CODE>
<DD>
A function to return the current end-of-file marker value.
<DT><CODE>herr_t (*read)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, void *buffer)</CODE>
<DD>
A function to read data from a file.
<DT><CODE>herr_t (*write)(H5FD_t *file, H5FD_mem_t type, hid_t dxpl, haddr_t addr, hsize_t size, const void *buffer)</CODE>
<DD>
A function to write data to a file.
<DT><CODE>herr_t (*flush)(H5FD_t *file)</CODE>
<DD>
A function which flushes cached data to the file.
<DT><CODE>H5FD_mem_t fl_map[H5FD_MEM_NTYPES]</CODE>
<DD>
An array which maps a file allocation request type to a free list.
</DL>
<P>
<STRONG>Example:</STRONG> The sec2 driver would be registered as:
</P>
<PRE>
static const H5FD_class_t H5FD_sec2_g = {
"sec2", /*name */
MAXADDR, /*maxaddr */
NULL, /*sb_size */
NULL, /*sb_encode */
NULL, /*sb_decode */
0, /*fapl_size */
NULL, /*fapl_get */
NULL, /*fapl_copy */
NULL, /*fapl_free */
0, /*dxpl_size */
NULL, /*dxpl_copy */
NULL, /*dxpl_free */
H5FD_sec2_open, /*open */
H5FD_sec2_close, /*close */
H5FD_sec2_cmp, /*cmp */
H5FD_sec2_query, /*query */
NULL, /*alloc */
NULL, /*free */
H5FD_sec2_get_eoa, /*get_eoa */
H5FD_sec2_set_eoa, /*set_eoa */
H5FD_sec2_get_eof, /*get_eof */
H5FD_sec2_read, /*read */
H5FD_sec2_write, /*write */
H5FD_sec2_flush, /*flush */
H5FD_FLMAP_SINGLE, /*fl_map */
};
hid_t
H5FD_sec2_init(void)
{
if (!H5FD_SEC2_g) {
H5FD_SEC2_g = H5FDregister(&#38;H5FD_sec2_g);
}
return H5FD_SEC2_g;
}
</PRE>
<P>
A driver can be removed from the library by unregistering it
</P>
<P>
<DL>
<DT><U>Function:</U> herr_t <B>H5Dunregister</B> <I>(hid_t <VAR>driver</VAR>)</I>
<DD><A NAME="IDX14"></A>
Where <VAR>driver</VAR> is the ID number returned when the driver was registered.
</DL>
</P>
<P>
Unregistering a driver makes it unusable for creating new file access or data
transfer property lists but doesn't affect any property lists or files that
already use that driver.
</P>
<H3><A NAME="SECProgNote" HREF="#TOCProgNote">Programming Note
for C++ Developers Using C Functions</A></H3>
<p>If a C routine that takes a function pointer as an argument is
called from within C++ code, the C routine should be returned from
normally. </p>
<p>Examples of this kind of routine include callbacks such as
<code>H5Pset_elink_cb</code> and <code>H5Pset_type_conv_cb</code>
and functions such as <code>H5Tconvert</code> and
<code>H5Ewalk2</code>.</p>
<p>Exiting the routine in its normal fashion allows the HDF5 C
Library to clean up its work properly. In other words, if the C++
application jumps out of the routine back to the C++
&ldquo;catch&rdquo; statement, the library is not given the
opportunity to close any temporary data structures that were set
up when the routine was called. The C++ application should save
some state as the routine is started so that any problem that
occurs might be diagnosed.</p>
<H2><A NAME="SEC24" HREF="#TOC24">Querying Driver Information</A></H2>
<P>
<DL>
<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fapl</VAR>)</I>
<DD><A NAME="IDX15"></A>
<DT><U>Function:</U> void * <B>H5Pget_driver_data</B> <I>(hid_t <VAR>fxpl</VAR>)</I>
<DD><A NAME="IDX16"></A>
</P>
<P>
This function is intended to be used by driver functions, not applications.
It returns a pointer directly into the file access property list
<CODE><VAR>fapl</VAR></CODE> which is a copy of the driver's file access mode originally
provided to the <CODE>H5Pset_driver</CODE> function. If its argument is a data
transfer property list <CODE>fxpl</CODE> then it returns a pointer to the
driver-specific data transfer information instead.
</DL>
</P>
<H1><A NAME="SEC25" HREF="#TOC25">Miscellaneous</A></H1>
<P>
The various private <CODE>H5F_low_*</CODE> functions will be replaced by public
<CODE>H5FD*</CODE> functions so they can be called from drivers.
</P>
<P>
All private functions <CODE>H5F_addr_*</CODE> which operate on addresses will be
renamed as public functions by removing the first underscore so they can be
called by drivers.
</P>
<P>
The <CODE>haddr_t</CODE> address data type will be passed by value throughout the
library. The original intent was that this type would eventually be a union of
file address types for the various drivers and may become quite large, but
that was back when drivers were part of HDF5. It will become an alias for an
unsigned integer type (32 or 64 bits depending on how the library was
configured).
</P>
<P>
The various <CODE>H5F*.c</CODE> driver files will be renamed <CODE>H5FD*.c</CODE> and each
will have a corresponding header file. All driver functions except the
initializer and API will be declared static.
</P>
<P>
This documentation didn't cover optimization functions which would be useful
to drivers like MPI-IO. Some drivers may be able to perform data pipeline
operations more efficiently than HDF5 and need to be given a chance to
override those parts of the pipeline. The pipeline would be designed to call
various H5FD optimization functions at various points which return one of
three values: the operation is not implemented by the driver, the operation is
implemented but failed in a non-recoverable manner, the operation is
implemented and succeeded.
</P>
<P>
Various parts of HDF5 check the only the top-level file driver and do
something special if it is the MPI-IO driver. However, we might want to be
able to put the MPI-IO driver under other drivers such as the raw part of a
split driver or under a debug driver whose sole purpose is to accumulate
statistics as it passes all requests through to the MPI-IO driver. Therefore
we will probably need a function which takes a format address and or object
type and returns the driver which would have been used at the lowest level to
process the request.
</P>
<P><HR><P>
<H1>Footnotes</H1>
<H3><A NAME="FOOT1" HREF="#DOCF1">(1)</A></H3>
<P>The driver name is by convention and might
not apply to drivers which are not distributed with HDF5.
<H3><A NAME="FOOT2" HREF="#DOCF2">(2)</A></H3>
<P>The access method also indicates how to translate
the storage name to a storage server such as a file, network protocol, or
memory.
<H3><A NAME="FOOT3" HREF="#DOCF3">(3)</A></H3>
<P>The term
"<EM>file</EM> access property list" is a misnomer since storage isn't
required to be a file.
<H3><A NAME="FOOT4" HREF="#DOCF4">(4)</A></H3>
<P>This
function is overloaded to operate on data transfer property lists also, as
described below.
<H3><A NAME="FOOT5" HREF="#DOCF5">(5)</A></H3>
<P>Read-only access is only appropriate when opening an existing
file.
<H3><A NAME="FOOT6" HREF="#DOCF6">(6)</A></H3>
<P>For instance, writing data to one handle will cause
the data to be immediately visible on the other handle.
<H3><A NAME="FOOT7" HREF="#DOCF7">(7)</A></H3>
<P>The ordering is
arbitrary as long as it's consistent within a particular file driver.
<H3><A NAME="FOOT8" HREF="#DOCF8">(8)</A></H3>
<P>File access modes do not describe data, but rather
describe how the HDF5 format address space is mapped to the underlying
file(s). Thus, in general the mapping must be known before the file superblock
can be read. However, the user usually knows enough about the mapping for the
superblock to be readable and once the superblock is read the library can fill
in the missing parts of the mapping.
<P><HR><P>
<?php include("../ed_libs/Footer2.htm"); ?>
</BODY>
</HTML>
doxygen/hdf5_footer.html
+21
View File
@@ -0,0 +1,21 @@
<!-- start footer part -->
<!--BEGIN GENERATE_TREEVIEW-->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
$navpath
<li class="footer">$generatedby
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="$relpath^doxygen.png" alt="doxygen"/></a> $doxygenversion </li>
</ul>
</div>
<!--END GENERATE_TREEVIEW-->
<!--BEGIN !GENERATE_TREEVIEW-->
<hr class="footer"/><address class="footer"><small>
$generatedby &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="$relpath^doxygen.png" alt="doxygen"/>
</a> $doxygenversion
</small></address>
<!--END !GENERATE_TREEVIEW-->
</body>
</html>
doxygen/hdf5_header.html
+61
View File
@@ -0,0 +1,61 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen $doxygenversion"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<!--BEGIN PROJECT_NAME--><title>$projectname: $title</title><!--END PROJECT_NAME-->
<!--BEGIN !PROJECT_NAME--><title>$title</title><!--END !PROJECT_NAME-->
<link href="$relpath^tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="$relpath^jquery.js"></script>
<script type="text/javascript" src="$relpath^dynsections.js"></script>
$treeview
$search
$mathjax
<link href="$relpath^$stylesheet" rel="stylesheet" type="text/css" />
<link href="$relpath$hdf5doxy.css" rel="stylesheet" type="text/css">
<!-- $extrastylesheet -->
<script type="text/javascript" src="$relpath$hdf5_navtree_hacks.js"></script>
</head>
<body>
<div style="background:#FFDDDD;font-size:120%;text-align:center;margin:0;padding:5px">Please, help us to better know about our user community by answering the following short survey: <a href="https://www.hdfgroup.org/">https://www.hdfgroup.org/</a></div>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<!--BEGIN TITLEAREA-->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<!--BEGIN PROJECT_LOGO-->
<td id="projectlogo"><img alt="Logo" src="$relpath^$projectlogo"/></td>
<!--END PROJECT_LOGO-->
<!--BEGIN PROJECT_NAME-->
<td id="projectalign" style="padding-left: 0.5em;">
<div id="projectname"><a href="https://www.hdfgroup.org">$projectname</a>
<!--BEGIN PROJECT_NUMBER-->&#160;<span id="projectnumber">$projectnumber</span><!--END PROJECT_NUMBER-->
</div>
<!--BEGIN PROJECT_BRIEF--><div id="projectbrief">$projectbrief</div><!--END PROJECT_BRIEF-->
</td>
<!--END PROJECT_NAME-->
<!--BEGIN !PROJECT_NAME-->
<!--BEGIN PROJECT_BRIEF-->
<td id="projectalign" style="padding-left: 0.5em;">
<div id="projectbrief">$projectbrief</div>
</td>
<!--END PROJECT_BRIEF-->
<!--END !PROJECT_NAME-->
<!--BEGIN DISABLE_INDEX-->
<!--BEGIN SEARCHENGINE-->
<td>$searchbox</td>
<!--END SEARCHENGINE-->
<!--END DISABLE_INDEX-->
</tr>
</tbody>
</table>
</div>
<!--END TITLEAREA-->
<!-- end header part -->
doxygen/hdf5_navtree_hacks.js
+246
View File
@@ -0,0 +1,246 @@
// generate a table of contents in the side-nav based on the h1/h2 tags of the current page.
function generate_autotoc() {
var headers = $("h1, h2");
if(headers.length > 1) {
var toc = $("#side-nav").append('<div id="nav-toc" class="toc"><h3>Table of contents</h3></div>');
toc = $("#nav-toc");
var footer = $("#nav-path");
var footerHeight = footer.height();
toc = toc.append('<ul></ul>');
toc = toc.find('ul');
var indices = new Array();
indices[0] = 0;
indices[1] = 0;
var h1counts = $("h1").length;
headers.each(function(i) {
var current = $(this);
var levelTag = current[0].tagName.charAt(1);
if(h1counts==0)
levelTag--;
var cur_id = current.attr("id");
indices[levelTag-1]+=1;
var prefix = indices[0];
if (levelTag >1) {
prefix+="."+indices[1];
}
// Uncomment to add number prefixes
// current.html(prefix + " " + current.html());
for(var l = levelTag; l < 2; ++l){
indices[l] = 0;
}
if(cur_id == undefined) {
current.attr('id', 'title' + i);
current.addClass('anchor');
toc.append("<li class='level" + levelTag + "'><a id='link" + i + "' href='#title" +
i + "' title='" + current.prop("tagName") + "'>" + current.text() + "</a></li>");
} else {
toc.append("<li class='level" + levelTag + "'><a id='" + cur_id + "' href='#title" +
i + "' title='" + current.prop("tagName") + "'>" + current.text() + "</a></li>");
}
});
resizeHeight();
}
}
var global_navtree_object;
// Overloaded to remove links to sections/subsections
function getNode(o, po)
{
po.childrenVisited = true;
var l = po.childrenData.length-1;
for (var i in po.childrenData) {
var nodeData = po.childrenData[i];
if((!nodeData[1]) || (nodeData[1].indexOf('#')==-1)) // <- we added this line
po.children[i] = newNode(o, po, nodeData[0], nodeData[1], nodeData[2], i==l);
}
}
// Overloaded to adjust the size of the navtree wrt the toc
function resizeHeight()
{
var header = $("#top");
var sidenav = $("#side-nav");
var content = $("#doc-content");
var navtree = $("#nav-tree");
var footer = $("#nav-path");
var toc = $("#nav-toc");
var headerHeight = header.outerHeight();
var footerHeight = footer.outerHeight();
var tocHeight = toc.height();
var windowHeight = $(window).height() - headerHeight - footerHeight;
content.css({height:windowHeight + "px"});
navtree.css({height:(windowHeight-tocHeight) + "px"});
sidenav.css({height:windowHeight + "px"});
}
// Overloaded to save the root node into global_navtree_object
function initNavTree(toroot,relpath)
{
var o = new Object();
global_navtree_object = o; // <- we added this line
o.toroot = toroot;
o.node = new Object();
o.node.li = document.getElementById("nav-tree-contents");
o.node.childrenData = NAVTREE;
o.node.children = new Array();
o.node.childrenUL = document.createElement("ul");
o.node.getChildrenUL = function() { return o.node.childrenUL; };
o.node.li.appendChild(o.node.childrenUL);
o.node.depth = 0;
o.node.relpath = relpath;
o.node.expanded = false;
o.node.isLast = true;
o.node.plus_img = document.createElement("img");
o.node.plus_img.src = relpath+"ftv2pnode.png";
o.node.plus_img.width = 16;
o.node.plus_img.height = 22;
if (localStorageSupported()) {
var navSync = $('#nav-sync');
if (cachedLink()) {
showSyncOff(navSync,relpath);
navSync.removeClass('sync');
} else {
showSyncOn(navSync,relpath);
}
navSync.click(function(){ toggleSyncButton(relpath); });
}
navTo(o,toroot,window.location.hash,relpath);
$(window).bind('hashchange', function(){
if (window.location.hash && window.location.hash.length>1){
var a;
if ($(location).attr('hash')){
var clslink=stripPath($(location).attr('pathname'))+':'+
$(location).attr('hash').substring(1);
a=$('.item a[class$="'+clslink+'"]');
}
if (a==null || !$(a).parent().parent().hasClass('selected')){
$('.item').removeClass('selected');
$('.item').removeAttr('id');
}
var link=stripPath2($(location).attr('pathname'));
navTo(o,link,$(location).attr('hash'),relpath);
} else if (!animationInProgress) {
$('#doc-content').scrollTop(0);
$('.item').removeClass('selected');
$('.item').removeAttr('id');
navTo(o,toroot,window.location.hash,relpath);
}
})
$(window).on("load", showRoot);
}
// return false if the the node has no children at all, or has only section/subsection children
function checkChildrenData(node) {
if (!(typeof(node.childrenData)==='string')) {
for (var i in node.childrenData) {
var url = node.childrenData[i][1];
if(url.indexOf("#")==-1)
return true;
}
return false;
}
return (node.childrenData);
}
// Modified to:
// 1 - remove the root node
// 2 - remove the section/subsection children
function createIndent(o,domNode,node,level)
{
var level=-2; // <- we replaced level=-1 by level=-2
var n = node;
while (n.parentNode) { level++; n=n.parentNode; }
if (checkChildrenData(node)) { // <- we modified this line to use checkChildrenData(node) instead of node.childrenData
var imgNode = document.createElement("span");
imgNode.className = 'arrow';
imgNode.style.paddingLeft=(16*level).toString()+'px';
imgNode.innerHTML=arrowRight;
node.plus_img = imgNode;
node.expandToggle = document.createElement("a");
node.expandToggle.href = "javascript:void(0)";
node.expandToggle.onclick = function() {
if (node.expanded) {
$(node.getChildrenUL()).slideUp("fast");
node.plus_img.innerHTML=arrowRight;
node.expanded = false;
} else {
expandNode(o, node, false, false);
}
}
node.expandToggle.appendChild(imgNode);
domNode.appendChild(node.expandToggle);
} else {
var span = document.createElement("span");
span.className = 'arrow';
span.style.width = 16*(level+1)+'px';
span.innerHTML = '&#160;';
domNode.appendChild(span);
}
}
// Overloaded to automatically expand the selected node
function selectAndHighlight(hash,n)
{
var a;
if (hash) {
var link=stripPath($(location).attr('pathname'))+':'+hash.substring(1);
a=$('.item a[class$="'+link+'"]');
}
if (a && a.length) {
a.parent().parent().addClass('selected');
a.parent().parent().attr('id','selected');
highlightAnchor();
} else if (n) {
$(n.itemDiv).addClass('selected');
$(n.itemDiv).attr('id','selected');
}
if ($('#nav-tree-contents .item:first').hasClass('selected')) {
$('#nav-sync').css('top','30px');
} else {
$('#nav-sync').css('top','5px');
}
expandNode(global_navtree_object, n, true, true); // <- we added this line
showRoot();
}
$(document).ready(function() {
generate_autotoc();
(function (){ // wait until the first "selected" element has been created
try {
// this line will triger an exception if there is no #selected element, i.e., before the tree structure is complete.
document.getElementById("selected").className = "item selected";
// ok, the default tree has been created, we can keep going...
// expand the "Chapters" node
if(window.location.href.indexOf('unsupported')==-1)
expandNode(global_navtree_object, global_navtree_object.node.children[0].children[2], true, true);
else
expandNode(global_navtree_object, global_navtree_object.node.children[0].children[1], true, true);
// Hide the root node "HDF5"
$(document.getElementsByClassName('index.html')[0]).parent().parent().css({display:"none"});
} catch (err) {
setTimeout(arguments.callee, 10);
}
})();
$(window).on("load", resizeHeight);
});
doxygen/hdf5doxy.css
+251
View File
@@ -0,0 +1,251 @@
/******** HDF5 specific CSS code ************/
/**** Styles removing elements ****/
/* remove the "modules|classes" link for module pages (they are already in the TOC) */
div.summary {
display:none;
}
/* remove */
div.contents hr {
display:none;
}
/**** ****/
p, dl.warning, dl.attention, dl.note
{
max-width:60em;
text-align:justify;
}
li {
max-width:55em;
text-align:justify;
}
img {
border: 0;
}
div.fragment {
display:table; /* this allows the element to be larger than its parent */
padding: 0pt;
}
pre.fragment {
border: 1px solid #cccccc;
margin: 2px 0px 2px 0px;
padding: 3px 5px 3px 5px;
}
/* Common style for all HDF5's tables */
table.example, table.manual, table.manual-vl, table.manual-hl {
max-width:100%;
border-collapse: collapse;
border-style: solid;
border-width: 1px;
border-color: #cccccc;
font-size: 1em;
box-shadow: 5px 5px 5px rgba(0, 0, 0, 0.15);
-moz-box-shadow: 5px 5px 5px rgba(0, 0, 0, 0.15);
-webkit-box-shadow: 5px 5px 5px rgba(0, 0, 0, 0.15);
}
table.example th, table.manual th, table.manual-vl th, table.manual-hl th {
padding: 0.5em 0.5em 0.5em 0.5em;
text-align: left;
padding-right: 1em;
color: #555555;
background-color: #F4F4E5;
background-image: -webkit-gradient(linear,center top,center bottom,from(#FFFFFF), color-stop(0.3,#FFFFFF), color-stop(0.30,#FFFFFF), color-stop(0.98,#F4F4E5), to(#ECECDE));
background-image: -moz-linear-gradient(center top, #FFFFFF 0%, #FFFFFF 30%, #F4F4E5 98%, #ECECDE);
filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#FFFFFF', endColorstr='#F4F4E5');
}
table.example td, table.manual td, table.manual-vl td, table.manual-hl td {
vertical-align:top;
border-width: 1px;
border-color: #cccccc;
}
/* header of headers */
table th.meta {
text-align:center;
font-size: 1.2em;
background-color:#FFFFFF;
}
/* intermediate header */
table th.inter {
text-align:left;
background-color:#FFFFFF;
background-image:none;
border-style:solid solid solid solid;
border-width: 1px;
border-color: #cccccc;
}
/** class for example / output tables **/
table.example {
}
table.example th {
}
table.example td {
padding: 0.5em 0.5em 0.5em 0.5em;
vertical-align:top;
}
/* standard class for the manual */
table.manual, table.manual-vl, table.manual-hl {
padding: 0.2em 0em 0.5em 0em;
}
table.manual th, table.manual-vl th, table.manual-hl th {
margin: 0em 0em 0.3em 0em;
}
table.manual td, table.manual-vl td, table.manual-hl td {
padding: 0.3em 0.5em 0.3em 0.5em;
vertical-align:top;
border-width: 1px;
}
table.manual td.alt, table.manual tr.alt, table.manual-vl td.alt, table.manual-vl tr.alt {
background-color: #F4F4E5;
}
table.manual-vl th, table.manual-vl td, table.manual-vl td.alt {
border-color: #cccccc;
border-width: 1px;
border-style: none solid none solid;
}
table.manual-vl th.inter {
border-style: solid solid solid solid;
}
table.manual-hl td {
border-color: #cccccc;
border-width: 1px;
border-style: solid none solid none;
}
table td.code {
font-family: monospace;
}
h2 {
margin-top:2em;
border-style: none none solid none;
border-width: 1px;
border-color: #cccccc;
}
/**** Table of content in the side-nav ****/
div.toc {
margin:0;
padding: 0.3em 0 0 0;
width:100%;
float:none;
position:absolute;
bottom:0;
border-radius:0px;
border-style: solid none none none;
max-height:50%;
overflow-y: scroll;
}
div.toc h3 {
margin-left: 0.5em;
margin-bottom: 0.2em;
}
div.toc ul {
margin: 0.2em 0 0.4em 0.5em;
}
span.cpp11,span.cpp14,span.cpp17 {
color: #119911;
font-weight: bold;
}
.newin3x {
color: #a37c1a;
font-weight: bold;
}
div.warningbox {
max-width:60em;
border-style: solid solid solid solid;
border-color: red;
border-width: 3px;
}
/**** old HDF5's styles ****/
table.tutorial_code td {
border-color: transparent; /* required for Firefox */
padding: 3pt 5pt 3pt 5pt;
vertical-align: top;
}
/* Whenever doxygen meets a '\n' or a '<BR/>', it will put
* the text containing the character into a <p class="starttd">.
* This little hack together with table.tutorial_code td.note
* aims at fixing this issue. */
table.tutorial_code td.note p.starttd {
margin: 0px;
border: none;
padding: 0px;
}
div.eimainmenu {
text-align: center;
}
/* center version number on main page */
h3.version {
text-align: center;
}
td.width20em p.endtd {
width: 20em;
}
/* needed for huge screens */
.ui-resizable-e {
background-repeat: repeat-y;
}
/* Style external links -- nav-tree is different */
#nav-tree .label a {
padding:2px 16px 2px 2px;
}
a {
outline: none;
text-decoration: none;
padding: 2px 1px 0;
}
a[href*="http"] {
background: url('https://mdn.mozillademos.org/files/12982/external-link-52.png') no-repeat 100% 0;
background-size: 12px 12px;
padding-right: 16px;
}
doxygen/hdf5doxy_layout.xml
+182
View File
@@ -0,0 +1,182 @@
<?xml version="1.0"?>
<doxygenlayout version="1.0">
<!-- Navigation index tabs for HTML output -->
<navindex>
<tab type="user" url="index.html" title="Overview" />
<tab type="user" url="https://portal.hdfgroup.org/display/HDF5/Learning+HDF5" title="Getting started" />
<tab type="user" url="@ref Cookbook" title="Cookbook" />
<tab type="user" url="https://portal.hdfgroup.org/display/HDF5/HDF5+User+Guides" title="User Guides" />
<tab type="user" url="https://portal.hdfgroup.org/display/HDF5/HDF5+Application+Developer%27s+Guide" title="Application Developer's Guide" />
<tab type="user" url="https://portal.hdfgroup.org/display/HDF5/HDF5+Glossary" title="Glossary" />
<tab type="user" url="@ref RM" title="Reference Manual" />
<tab type="user" url="@ref TN" title="Technical Notes" />
<tab type="user" url="@ref SPEC" title="Specifications" />
<tab type="user" url="@ref About" title="About" />
</navindex>
<!-- Layout definition for a class page -->
<class>
<briefdescription visible="no"/>
<includes visible="$SHOW_INCLUDE_FILES"/>
<detaileddescription title=""/>
<inheritancegraph visible="$CLASS_GRAPH"/>
<collaborationgraph visible="$COLLABORATION_GRAPH"/>
<allmemberslink visible="yes"/>
<memberdecl>
<nestedclasses visible="yes" title=""/>
<publictypes title=""/>
<publicslots title=""/>
<signals title=""/>
<publicmethods title=""/>
<publicstaticmethods title=""/>
<publicattributes title=""/>
<publicstaticattributes title=""/>
<protectedtypes title=""/>
<protectedslots title=""/>
<protectedmethods title=""/>
<protectedstaticmethods title=""/>
<protectedattributes title=""/>
<protectedstaticattributes title=""/>
<packagetypes title=""/>
<packagemethods title=""/>
<packagestaticmethods title=""/>
<packageattributes title=""/>
<packagestaticattributes title=""/>
<properties title=""/>
<events title=""/>
<privatetypes title=""/>
<privateslots title=""/>
<privatemethods title=""/>
<privatestaticmethods title=""/>
<privateattributes title=""/>
<privatestaticattributes title=""/>
<friends title=""/>
<related title="" subtitle=""/>
<membergroups visible="yes"/>
</memberdecl>
<memberdef>
<inlineclasses title=""/>
<typedefs title=""/>
<enums title=""/>
<constructors title=""/>
<functions title=""/>
<related title=""/>
<variables title=""/>
<properties title=""/>
<events title=""/>
</memberdef>
<usedfiles visible="$SHOW_USED_FILES"/>
<authorsection visible="yes"/>
</class>
<!-- Layout definition for a namespace page -->
<namespace>
<briefdescription visible="yes"/>
<memberdecl>
<nestednamespaces visible="yes" title=""/>
<classes visible="yes" title=""/>
<typedefs title=""/>
<enums title=""/>
<functions title=""/>
<variables title=""/>
<membergroups visible="yes"/>
</memberdecl>
<detaileddescription title=""/>
<memberdef>
<inlineclasses title=""/>
<typedefs title=""/>
<enums title=""/>
<functions title=""/>
<variables title=""/>
</memberdef>
<authorsection visible="yes"/>
</namespace>
<!-- Layout definition for a file page -->
<file>
<briefdescription visible="yes"/>
<includes visible="$SHOW_INCLUDE_FILES"/>
<includegraph visible="$INCLUDE_GRAPH"/>
<includedbygraph visible="$INCLUDED_BY_GRAPH"/>
<sourcelink visible="yes"/>
<memberdecl>
<classes visible="yes" title=""/>
<namespaces visible="yes" title=""/>
<defines title=""/>
<typedefs title=""/>
<enums title=""/>
<functions title=""/>
<variables title=""/>
<membergroups visible="yes"/>
</memberdecl>
<detaileddescription title=""/>
<memberdef>
<inlineclasses title=""/>
<defines title=""/>
<typedefs title=""/>
<enums title=""/>
<functions title=""/>
<variables title=""/>
</memberdef>
<authorsection/>
</file>
<!-- Layout definition for a group page -->
<group>
<briefdescription visible="no"/>
<detaileddescription title=""/>
<groupgraph visible="$GROUP_GRAPHS"/>
<memberdecl>
<nestedgroups visible="yes" title=""/>
<dirs visible="yes" title=""/>
<files visible="yes" title=""/>
<namespaces visible="yes" title=""/>
<classes visible="yes" title=""/>
<defines title=""/>
<typedefs title=""/>
<enums title=""/>
<enumvalues title=""/>
<functions title=""/>
<variables title=""/>
<signals title=""/>
<publicslots title=""/>
<protectedslots title=""/>
<privateslots title=""/>
<events title=""/>
<properties title=""/>
<friends title=""/>
<membergroups visible="yes"/>
</memberdecl>
<memberdef>
<pagedocs/>
<inlineclasses title=""/>
<defines title=""/>
<typedefs title=""/>
<enums title=""/>
<enumvalues title=""/>
<functions title=""/>
<variables title=""/>
<signals title=""/>
<publicslots title=""/>
<protectedslots title=""/>
<privateslots title=""/>
<events title=""/>
<properties title=""/>
<friends title=""/>
</memberdef>
<authorsection visible="yes"/>
</group>
<!-- Layout definition for a directory page -->
<directory>
<briefdescription visible="yes"/>
<directorygraph visible="yes"/>
<memberdecl>
<dirs visible="yes"/>
<files visible="yes"/>
</memberdecl>
<detaileddescription title=""/>
</directory>
</doxygenlayout>
doxygen/img/FF-IH_FileGroup.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.3 KiB

doxygen/img/FF-IH_FileObject.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.1 KiB

doxygen/img/FileFormatSpecChunkDiagram.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

doxygen/img/HDFG-logo.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.4 KiB

After

Width:  |  Height:  |  Size: 1.6 KiB

doxygen/img/PaletteExample1.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.7 KiB

doxygen/img/Palettes.fm.anc.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.6 KiB

doxygen/img/ftv2node.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 86 B

doxygen/img/ftv2pnode.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 229 B

src/CMakeLists.txt
+1
View File
@@ -1378,6 +1378,7 @@ if (DOXYGEN_FOUND)
set (DOXYGEN_OPTIMIZE_OUTPUT_FOR_C YES)
set (DOXYGEN_MACRO_EXPANSION YES)
set (DOXYGEN_OUTPUT_DIRECTORY ${HDF5_BINARY_DIR}/hdf5lib_docs)
set (DOXYGEN_EXAMPLES_DIRECTORY ${HDF5_DOXYGEN_DIR}/examples)
# This configure and custom target work together
# Replace variables inside @@ with the current values
src/H5ACpublic.h
+282 -59
View File
@@ -442,124 +442,347 @@ extern "C" {
#define H5AC_METADATA_WRITE_STRATEGY__PROCESS_0_ONLY 0
#define H5AC_METADATA_WRITE_STRATEGY__DISTRIBUTED 1
/**
* H5AC_cache_config_t is a public structure intended for use in public APIs.
* At least in its initial incarnation, it is basically a copy of \c struct
* \c H5C_auto_size_ctl_t, minus the \c report_fcn field, and plus the
* \c dirty_bytes_threshold field.
*
* The \c report_fcn field is omitted, as including it would require us to make
* \c H5C_t structure public.
*
* The \c dirty_bytes_threshold field does not appear in \c H5C_auto_size_ctl_t,
* as synchronization between caches on different processes is handled at the \c
* H5AC level, not at the level of \c H5C. Note however that there is
* considerable interaction between this value and the other fields in this
* structure.
*
* Similarly, the \c open_trace_file, \c close_trace_file, and \c
* trace_file_name fields do not appear in \c H5C_auto_size_ctl_t, as most trace
* file issues are handled at the \c H5AC level. The one exception is storage
* of the pointer to the trace file, which is handled by \c H5C.
*
* The structure is in H5ACpublic.h as we may wish to allow different
* configuration options for metadata and raw data caches.
*/
//! <!-- [H5AC_cache_config_t_snip] -->
typedef struct H5AC_cache_config_t {
/* general configuration fields: */
//! <!-- [H5AC_cache_config_t_general_snip] -->
int version;
/**< Integer field indicating the the version of the H5AC_cache_config_t
* in use. This field should be set to #H5AC__CURR_CACHE_CONFIG_VERSION
* (defined in H5ACpublic.h). */
hbool_t rpt_fcn_enabled;
/**< Boolean flag indicating whether the adaptive cache resize report
* function is enabled. This field should almost always be set to disabled
* (0). Since resize algorithm activity is reported via stdout, it MUST be
* set to disabled (0) on Windows machines.\n
* The report function is not supported code, and can be expected to change
* between versions of the library. Use it at your own risk. */
hbool_t open_trace_file;
/**< Boolean field indicating whether the
* \ref H5AC_cache_config_t.trace_file_name "trace_file_name"
* field should be used to open a trace file for the cache.\n
* The trace file is a debugging feature that allows the capture
* of top level metadata cache requests for purposes of debugging
* and/or optimization. This field should normally be set to 0, as
* trace file collection imposes considerable overhead.\n
* This field should only be set to 1 when the
* \ref H5AC_cache_config_t.trace_file_name "trace_file_name"
* contains the full path of the desired trace file, and either
* there is no open trace file on the cache, or the
* \ref H5AC_cache_config_t.close_trace_file "close_trace_file"
* field is also 1.\n
* The trace file feature is unsupported unless used at the
* direction of The HDF Group. It is intended to allow The HDF
* Group to collect a trace of cache activity in cases of occult
* failures and/or poor performance seen in the field, so as to aid
* in reproduction in the lab. If you use it absent the direction
* of The HDF Group, you are on your own. */
hbool_t close_trace_file;
char trace_file_name[H5AC__MAX_TRACE_FILE_NAME_LEN + 1];
/**< Boolean field indicating whether the current trace file
*(if any) should be closed.\n
* See the above comments on the \ref H5AC_cache_config_t.open_trace_file
* "open_trace_file" field. This field should be set to 0 unless there is
* an open trace file on the cache that you wish to close.\n
* The trace file feature is unsupported unless used at the direction of
* The HDF Group. It is intended to allow The HDF Group to collect a trace
* of cache activity in cases of occult failures and/or poor performance
* seen in the field, so as to aid in reproduction in the lab. If you use
* it absent the direction of The HDF Group, you are on your own. */
char trace_file_name[H5AC__MAX_TRACE_FILE_NAME_LEN + 1];
/**< Full path of the trace file to be opened if the
* \ref H5AC_cache_config_t.open_trace_file "open_trace_file" field is set
* to 1.\n
* In the parallel case, an ascii representation of the MPI rank of the
* process will be appended to the file name to yield a unique trace file
* name for each process.\n
* The length of the path must not exceed #H5AC__MAX_TRACE_FILE_NAME_LEN
* characters.\n
* The trace file feature is unsupported unless used at the direction of
* The HDF Group. It is intended to allow The HDF Group to collect a trace
* of cache activity in cases of occult failures and/or poor performance
* seen in the field, so as to aid in reproduction in the lab. If you use
* it absent the direction of The HDF Group, you are on your own. */
hbool_t evictions_enabled;
/**< A boolean flag indicating whether evictions from the metadata cache
* are enabled. This flag is initially set to enabled (1).\n
* In rare circumstances, the raw data throughput quirements may be so high
* that the user wishes to postpone metadata writes so as to reserve I/O
* throughput for raw data. The \p evictions_enabled field exists to allow
* this. However, this is an extreme step, and you have no business doing
* it unless you have read the User Guide section on metadata caching, and
* have considered all other options carefully.\n
* The \p evictions_enabled field may not be set to disabled (0)
* unless all adaptive cache resizing code is disabled via the
* \ref H5AC_cache_config_t.incr_mode "incr_mode",
* \ref H5AC_cache_config_t.flash_incr_mode "flash_incr_mode",
* \ref H5AC_cache_config_t.decr_mode "decr_mode" fields.\n
* When this flag is set to disabled (\c 0), the metadata cache will not
* attempt to evict entries to make space for new entries, and thus will
* grow without bound.\n
* Evictions will be re-enabled when this field is set back to \c 1.
* This should be done as soon as possible. */
hbool_t set_initial_size;
size_t initial_size;
/**< Boolean flag indicating whether the cache should be created
* with a user specified initial size. */
size_t initial_size;
/**< If \ref H5AC_cache_config_t.set_initial_size "set_initial_size"
* is set to 1, \p initial_size must contain he desired initial size in
* bytes. This value must lie in the closed interval
* [ \p min_size, \p max_size ]. (see below) */
double min_clean_fraction;
/**< This field specifies the minimum fraction of the cache
* that must be kept either clean or empty.\n
* The value must lie in the interval [0.0, 1.0]. 0.01 is a good place to
* start in the serial case. In the parallel case, a larger value is needed
* -- see the overview of the metadata cache in the
* “Metadata Caching in HDF5” section of the -- <em>HDF5 User’s Guide</em>
* for details. */
size_t max_size;
/**< Upper bound (in bytes) on the range of values that the
* adaptive cache resize code can select as the maximum cache size. */
size_t min_size;
/**< Lower bound (in bytes) on the range of values that the
* adaptive cache resize code can select as the mininum cache * size. */
long int epoch_length;
/**< Number of cache accesses between runs of the adaptive cache resize
* code. 50,000 is a good starting number. */
//! <!-- [H5AC_cache_config_t_general_snip] -->
/* size increase control fields: */
//! <!-- [H5AC_cache_config_t_incr_snip] -->
enum H5C_cache_incr_mode incr_mode;
/**< Enumerated value indicating the operational mode of the automatic
* cache size increase code. At present, only two values listed in
* #H5C_cache_incr_mode are legal. */
double lower_hr_threshold;
/**< Hit rate threshold used by the hit rate threshold cache size
* increment algorithm.\n
* When the hit rate over an epoch is below this threshold and the cache
* is full, the maximum size of the cache is multiplied by increment
* (below), and then clipped as necessary to stay within \p max_size, and
* possibly \p max_increment.\n
* This field must lie in the interval [0.0, 1.0]. 0.8 or 0.9 is a good
* place to start. */
double increment;
/**< Factor by which the hit rate threshold cache size increment
* algorithm multiplies the current cache max size to obtain a tentative
* new cache size.\n
* The actual cache size increase will be clipped to satisfy the \p max_size
* specified in the general configuration, and possibly max_increment
* below.\n
* The parameter must be greater than or equal to 1.0 -- 2.0 is a reasonable
* value.\n
* If you set it to 1.0, you will effectively disable cache size increases.
*/
hbool_t apply_max_increment;
size_t max_increment;
/**< Boolean flag indicating whether an upper limit should be applied to
* the size of cache size increases. */
size_t max_increment;
/**< Maximum number of bytes by which cache size can be increased in a
* single step -- if applicable. */
enum H5C_cache_flash_incr_mode flash_incr_mode;
double flash_multiple;
double flash_threshold;
/**< Enumerated value indicating the operational mode of the flash cache
* size increase code. At present, only two listed values in
* #H5C_cache_flash_incr_mode are legal.*/
double flash_multiple;
/**< The factor by which the size of the triggering entry / entry size
* increase is multiplied to obtain the initial cache size increment. This
* increment may be reduced to reflect existing free space in the cache and
* the \p max_size field above.\n
* The parameter must lie in the interval [0.0, 1.0]. 0.1 or 0.05 is a good
* place to start.\n
* At present, this field must lie in the range [0.1, 10.0]. */
double flash_threshold;
/**< The factor by which the current maximum cache size is multiplied to
* obtain the minimum size entry / entry size increase which may trigger a
* flash cache size increase. \n
* At present, this value must lie in the range [0.1, 1.0]. */
//! <!-- [H5AC_cache_config_t_incr_snip] -->
/* size decrease control fields: */
//! <!-- [H5AC_cache_config_t_decr_snip] -->
enum H5C_cache_decr_mode decr_mode;
/**< Enumerated value indicating the operational mode of the tomatic
* cache size decrease code. At present, the values listed in
* #H5C_cache_decr_mode are legal.*/
double upper_hr_threshold;
/**< Hit rate threshold for the hit rate threshold and ageout with hit
* rate threshold cache size decrement algorithms.\n
* When \p decr_mode is #H5C_decr__threshold, and the hit rate over a given
* epoch exceeds the supplied threshold, the current maximum cache
* size is multiplied by decrement to obtain a tentative new (and smaller)
* maximum cache size.\n
* When \p decr_mode is #H5C_decr__age_out_with_threshold, there is
* no attempt to find and evict aged out entries unless the hit rate in
* the previous epoch exceeded the supplied threshold.\n
* This field must lie in the interval [0.0, 1.0].\n
* For #H5C_incr__threshold, .9995 or .99995 is a good place to start.\n
* For #H5C_decr__age_out_with_threshold, .999 might be more useful.*/
double decrement;
/**< In the hit rate threshold cache size decrease algorithm, this
* parameter contains the factor by which the current max cache size is
* multiplied to produce a tentative new cache size.\n
* The actual cache size decrease will be clipped to satisfy the
* \ref H5AC_cache_config_t.min_size "min_size" specified in the general
* configuration, and possibly \ref H5AC_cache_config_t.max_decrement
* "max_decrement".\n
* The parameter must be be in the interval [0.0, 1.0].\n
* If you set it to 1.0, you will effectively
* disable cache size decreases. 0.9 is a reasonable starting point. */
hbool_t apply_max_decrement;
size_t max_decrement;
/**< Boolean flag indicating ether an upper limit should be applied to
* the size of cache size decreases. */
size_t max_decrement;
/**< Maximum number of bytes by which the maximum cache size can be
* decreased in any single step -- if applicable.*/
int epochs_before_eviction;
/**< In the ageout based cache size reduction algorithms, this field
* contains the minimum number of epochs an entry must remain unaccessed in
* cache before the cache size reduction algorithm tries to evict it. 3 is a
* reasonable value. */
hbool_t apply_empty_reserve;
double empty_reserve;
/**< Boolean flag indicating whether the ageout based decrement
* algorithms will maintain a empty reserve when decreasing cache size. */
double empty_reserve;
/**< Empty reserve as a fraction maximum cache size if applicable.\n When
* so directed, the ageout based algorithms will not decrease the maximum
* cache size unless the empty reserve can be met.\n The parameter must lie
* in the interval [0.0, 1.0]. 0.1 or 0.05 is a good place to start. */
//! <!-- [H5AC_cache_config_t_decr_snip] -->
/* parallel configuration fields: */
//! <!-- [H5AC_cache_config_t_parallel_snip] -->
size_t dirty_bytes_threshold;
int metadata_write_strategy;
/**< Threshold number of bytes of dirty metadata generation for
* triggering synchronizations of the metadata caches serving the target
* file in the parallel case.\n Synchronization occurs whenever the number
* of bytes of dirty metadata created since the last synchronization exceeds
* this limit.\n This field only applies to the parallel case. While it is
* ignored elsewhere, it can still draw a value out of bounds error.\n It
* must be consistant across all caches on any given file.\n By default,
* this field is set to 256 KB. It shouldn't be more than half the current
* max cache size times the min clean fraction. */
int metadata_write_strategy;
/**< Desired metadata write strategy. The valid values for this field
* are:\n #H5AC_METADATA_WRITE_STRATEGY__PROCESS_0_ONLY: Specifies tha only
* process zero is allowed to write dirty metadata to disk.\n
* #H5AC_METADATA_WRITE_STRATEGY__DISTRIBUTED: Specifies that process zero
* still makes the decisions as to what entries should be flushed, but the
* actual flushes are distributed across the processes in the computation to
* the extent possible.\n The src/H5ACpublic.h include file in the HDF5
* library has detailed information on each strategy. */
//! <!-- [H5AC_cache_config_t_parallel_snip] -->
} H5AC_cache_config_t;
/****************************************************************************
*
* structure H5AC_cache_image_config_t
*
* H5AC_cache_image_ctl_t is a public structure intended for use in public
* APIs. At least in its initial incarnation, it is a copy of struct
* H5C_cache_image_ctl_t.
*
* The fields of the structure are discussed individually below:
*
* version: Integer field containing the version number of this version
* of the H5C_image_ctl_t structure. Any instance of
* H5C_image_ctl_t passed to the cache must have a known
* version number, or an error will be flagged.
*
* generate_image: Boolean flag indicating whether a cache image should
* be created on file close.
*
* save_resize_status: Boolean flag indicating whether the cache image
* should include the adaptive cache resize configuration and status.
* Note that this field is ignored at present.
*
* entry_ageout: Integer field indicating the maximum number of
* times a prefetched entry can appear in subsequent cache images.
* This field exists to allow the user to avoid the buildup of
* infrequently used entries in long sequences of cache images.
*
* The value of this field must lie in the range
* H5AC__CACHE_IMAGE__ENTRY_AGEOUT__NONE (-1) to
* H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX (100).
*
* H5AC__CACHE_IMAGE__ENTRY_AGEOUT__NONE means that no limit
* is imposed on number of times a prefeteched entry can appear
* in subsequent cache images.
*
* A value of 0 prevents prefetched entries from being included
* in cache images.
*
* Positive integers restrict prefetched entries to the specified
* number of appearances.
*
* Note that the number of subsequent cache images that a prefetched
* entry has appeared in is tracked in an 8 bit field. Thus, while
* H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX can be increased from its
* current value, any value in excess of 255 will be the functional
* equivalent of H5AC__CACHE_IMAGE__ENTRY_AGEOUT__NONE.
*
****************************************************************************/
//! <!-- [H5AC_cache_config_t_snip] -->
#define H5AC__CURR_CACHE_IMAGE_CONFIG_VERSION 1
#define H5AC__CACHE_IMAGE__ENTRY_AGEOUT__NONE -1
#define H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX 100
//! <!-- [H5AC_cache_image_config_t_snip] -->
/**
* H5AC_cache_image_config_t is a public structure intended for use in public
* APIs. At least in its initial incarnation, it is a copy of \c struct \c
* H5C_cache_image_ctl_t.
*/
typedef struct H5AC_cache_image_config_t {
int version;
int version;
/**< Integer field containing the version number of this version of the \c
* H5C_image_ctl_t structure. Any instance of \c H5C_image_ctl_t passed
* to the cache must have a known version number, or an error will be
* flagged.
*/
hbool_t generate_image;
/**< Boolean flag indicating whether a cache image should be created on file
* close.
*/
hbool_t save_resize_status;
int entry_ageout;
/**< Boolean flag indicating whether the cache image should include the
* adaptive cache resize configuration and status. Note that this field
* is ignored at present.
*/
int entry_ageout;
/**< Integer field indicating the maximum number of times a
* prefetched entry can appear in subsequent cache images. This field
* exists to allow the user to avoid the buildup of infrequently used
* entries in long sequences of cache images.
*
* The value of this field must lie in the range \ref
* H5AC__CACHE_IMAGE__ENTRY_AGEOUT__NONE (-1) to \ref
* H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX (100).
*
* \ref H5AC__CACHE_IMAGE__ENTRY_AGEOUT__NONE means that no limit is
* imposed on number of times a prefeteched entry can appear in subsequent
* cache images.
*
* A value of 0 prevents prefetched entries from being included in cache
* images.
*
* Positive integers restrict prefetched entries to the specified number
* of appearances.
*
* Note that the number of subsequent cache images that a prefetched entry
* has appeared in is tracked in an 8 bit field. Thus, while \ref
* H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX can be increased from its current
* value, any value in excess of 255 will be the functional equivalent of
* \ref H5AC__CACHE_IMAGE__ENTRY_AGEOUT__NONE.
*/
} H5AC_cache_image_config_t;
//! <!-- [H5AC_cache_image_config_t_snip] -->
#ifdef __cplusplus
}
#endif
src/H5Amodule.h
+30 -10
View File
@@ -30,19 +30,39 @@
#define H5_MY_PKG_INIT YES
/**\defgroup H5A H5A
* \brief Attribute Interface
*
* \details The Attribute Interface, H5A, provides a mechanism for attaching
* additional information to a dataset, group, or named datatype.
* Use the functions in this module to manage HDF5 attributes.
*
* Attributes are accessed by opening the object that they are
* attached to and are not independent objects. Typically an
* attribute is small in size and contains user metadata about the
* object that it is attached to.
* The Attribute Interface, H5A, provides a mechanism for attaching additional
* information to a dataset, group, or named datatype.
*
* Attributes look similar to HDF5 datasets in that they have a
* datatype and dataspace. However, they do not support partial
* I/O operations and cannot be compressed or extended.
* Attributes are accessed by opening the object that they are attached to and
* are not independent objects. Typically an attribute is small in size and
* contains user metadata about the object that it is attached to.
*
* Attributes look similar to HDF5 datasets in that they have a datatype and
* dataspace. However, they do not support partial I/O operations and cannot be
* compressed or extended.
*
* <table>
* <tr><th>Create</th><th>Read</th></tr>
* <tr valign="top">
* <td>
* \snippet H5A_examples.c create
* </td>
* <td>
* \snippet H5A_examples.c read
* </td>
* <tr><th>Update</th><th>Delete</th></tr>
* <tr valign="top">
* <td>
* \snippet H5A_examples.c update
* </td>
* <td>
* \snippet H5A_examples.c delete
* </td>
* </tr>
* </table>
*
*/
src/H5Apublic.h
+126 -67
View File
@@ -22,19 +22,40 @@
#include "H5Opublic.h" /* Object Headers */
#include "H5Tpublic.h" /* Datatypes */
/* Information struct for attribute (for H5Aget_info/H5Aget_info_by_idx) */
//! [H5A_info_t_snip]
//! <!-- [H5A_info_t_snip] -->
/**
* Information struct for H5Aget_info() / H5Aget_info_by_idx()
*/
typedef struct {
hbool_t corder_valid; /* Indicate if creation order is valid */
H5O_msg_crt_idx_t corder; /* Creation order */
H5T_cset_t cset; /* Character set of attribute name */
hsize_t data_size; /* Size of raw data */
hbool_t corder_valid; /**< Indicate if creation order is valid */
H5O_msg_crt_idx_t corder; /**< Creation order */
H5T_cset_t cset; /**< Character set of attribute name */
hsize_t data_size; /**< Size of raw data */
} H5A_info_t;
//! [H5A_info_t_snip]
//! <!-- [H5A_info_t_snip] -->
/* Typedef for H5Aiterate2() callbacks */
//! <!-- [H5A_operator2_t_snip] -->
/**
* Typedef for H5Aiterate2() / H5Aiterate_by_name() callbacks
* \param[in] location_id The identifier for the group, dataset
* or named datatype being iterated over
* \param[in] attr_name The name of the current object attribute
* \param[in] ainfo The attribute’s info struct
* \param[in,out] op_data A pointer to the operator data passed in to
* H5Aiterate2() or H5Aiterate_by_name()
* \returns The return values from an operator are:
* \li Zero causes the iterator to continue, returning zero when
* all attributes have been processed.
* \li Positive causes the iterator to immediately return that
* positive value, indicating short-circuit success. The
* iterator can be restarted at the next attribute.
* \li Negative causes the iterator to immediately return that value,
* indicating failure. The iterator can be restarted at the next
* attribute.
*/
typedef herr_t (*H5A_operator2_t)(hid_t location_id /*in*/, const char *attr_name /*in*/,
const H5A_info_t *ainfo /*in*/, void *op_data /*in,out*/);
//! <!-- [H5A_operator2_t_snip] -->
/********************/
/* Public Variables */
@@ -105,8 +126,8 @@ H5_DLL herr_t H5Aclose(hid_t attr_id);
* The attribute identifier returned by this function must be released
* with H5Aclose() resource leaks will develop.
*
* \note The \p acpl and \p aapl parameters are currently not used; specify
* #H5P_DEFAULT.
* \note The \p aapl parameter is currently not used; specify #H5P_DEFAULT.
*
* \note If \p loc_id is a file identifier, the attribute will be attached
* that file’s root group.
*
@@ -117,6 +138,11 @@ H5_DLL herr_t H5Aclose(hid_t attr_id);
*/
H5_DLL hid_t H5Acreate2(hid_t loc_id, const char *attr_name, hid_t type_id, hid_t space_id, hid_t acpl_id,
hid_t aapl_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Acreate}
*/
H5_DLL hid_t H5Acreate_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *attr_name, hid_t type_id, hid_t space_id, hid_t acpl_id,
hid_t aapl_id, hid_t es_id);
@@ -659,30 +685,12 @@ H5_DLL hid_t H5Aget_type(hid_t attr_id);
* the value returned identifies the parameter to be operated on
* in the next step of the iteration.
*
* The #H5A_operator2_t prototype for the \p op parameter is a
* user defined function where:
* The operation receives the location identifier for the group or
* dataset being iterated over, \p location_id; the name of the
* current object attribute, \p attr_name; the attribute’s info
* struct, \p ainfo; and a pointer to the operator data passed
* into H5Aiterate2(), \p op_data.
*
* Valid return values from an operator and the resulting
* H5Aiterate2() and \p op behavior are as follows:
*
* \li Zero causes the iterator to continue, returning zero when
* all attributes have been processed.
* \li A positive value causes the iterator to immediately return
* that positive value, indicating short-circuit success. The
* iterator can be restarted at the next attribute, as
* indicated by the return value of \p idx.
* \li A negative value causes the iterator to immediately return
* that value, indicating failure. The iterator can be
* restarted at the next attribute, as indicated by the return
* value of \p idx.
* \p op is a user-defined function whose prototype is defined
* as follows:
* \snippet this H5A_operator2_t_snip
* \click4more
*
* \note This function is also available through the H5Aiterate() macro.
* \todo Add snippet for H5A_operator2_t
*
* \since 1.8.0
*
@@ -751,13 +759,10 @@ H5_DLL herr_t H5Aiterate2(hid_t loc_id, H5_index_t idx_type, H5_iter_order_t ord
* the value returned identifies the parameter to be operated on in
* the next step of the iteration.
*
* The #H5A_operator2_t prototype for the \p op parameter is a
* user defined function where:
* The operation receives the location identifier for the group or
* dataset being iterated over, \p location_id; the name of the
* current object attribute, \p attr_name; the attribute’s info
* struct, \p ainfo; and a pointer to the operator data passed
* into H5Aiterate_by_name(), \p op_data.
* \p op is a user-defined function whose prototype is defined
* as follows:
* \snippet this H5A_operator2_t_snip
* \click4more
*
* Valid return values from an operator and the resulting
* H5Aiterate_by_name() and \p op behavior are as follows:
@@ -777,17 +782,21 @@ H5_DLL herr_t H5Aiterate2(hid_t loc_id, H5_index_t idx_type, H5_iter_order_t ord
* information regarding the properties of links required to access
* the object, \p obj_name.
*
* \todo Add snippet to show H5Aoperator2_t.
* \since 1.8.0
*
*/
H5_DLL herr_t H5Aiterate_by_name(hid_t loc_id, const char *obj_name, H5_index_t idx_type,
H5_iter_order_t order, hsize_t *idx, H5A_operator2_t op, void *op_data,
hid_t lapl_id);
H5_DLL hid_t H5Acreate_by_name_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *obj_name, const char *attr_name, hid_t type_id,
hid_t space_id, hid_t acpl_id, hid_t aapl_id, hid_t lapl_id,
hid_t es_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Acreate_by_name}
*/
H5_DLL hid_t H5Acreate_by_name_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *obj_name, const char *attr_name, hid_t type_id,
hid_t space_id, hid_t acpl_id, hid_t aapl_id, hid_t lapl_id,
hid_t es_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup H5A
@@ -819,6 +828,11 @@ H5_DLL hid_t H5Acreate_by_name_async(const char *app_file, const char *app_func
* \see H5Aclose(), H5Acreate()
*/
H5_DLL hid_t H5Aopen(hid_t obj_id, const char *attr_name, hid_t aapl_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Aopen}
*/
H5_DLL hid_t H5Aopen_async(const char *app_file, const char *app_func, unsigned app_line, hid_t obj_id,
const char *attr_name, hid_t aapl_id, hid_t es_id);
/*--------------------------------------------------------------------------*/
@@ -868,6 +882,11 @@ H5_DLL hid_t H5Aopen_async(const char *app_file, const char *app_func, unsigned
*/
H5_DLL hid_t H5Aopen_by_idx(hid_t loc_id, const char *obj_name, H5_index_t idx_type, H5_iter_order_t order,
hsize_t n, hid_t aapl_id, hid_t lapl_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Aopen_by_idx}
*/
H5_DLL hid_t H5Aopen_by_idx_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *obj_name, H5_index_t idx_type, H5_iter_order_t order, hsize_t n,
hid_t aapl_id, hid_t lapl_id, hid_t es_id);
@@ -914,6 +933,11 @@ H5_DLL hid_t H5Aopen_by_idx_async(const char *app_file, const char *app_func, un
*/
H5_DLL hid_t H5Aopen_by_name(hid_t loc_id, const char *obj_name, const char *attr_name, hid_t aapl_id,
hid_t lapl_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Aopen_by_name}
*/
H5_DLL hid_t H5Aopen_by_name_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *obj_name, const char *attr_name, hid_t aapl_id,
hid_t lapl_id, hid_t es_id);
@@ -967,6 +991,11 @@ H5_DLL herr_t H5Aread(hid_t attr_id, hid_t type_id, void *buf);
*
*/
H5_DLL herr_t H5Arename(hid_t loc_id, const char *old_name, const char *new_name);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Aread}
*/
H5_DLL herr_t H5Aread_async(const char *app_file, const char *app_func, unsigned app_line, hid_t attr_id,
hid_t dtype_id, void *buf, hid_t es_id);
/*--------------------------------------------------------------------------*/
@@ -1001,15 +1030,40 @@ H5_DLL herr_t H5Aread_async(const char *app_file, const char *app_func, unsigned
*
*/
H5_DLL herr_t H5Awrite(hid_t attr_id, hid_t type_id, const void *buf);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Awrite}
*/
H5_DLL herr_t H5Awrite_async(const char *app_file, const char *app_func, unsigned app_line, hid_t attr_id,
hid_t type_id, const void *buf, hid_t es_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Arename}
*/
H5_DLL herr_t H5Arename_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *old_name, const char *new_name, hid_t es_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Arename_by_name}
*/
H5_DLL herr_t H5Arename_by_name_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *obj_name, const char *old_attr_name,
const char *new_attr_name, hid_t lapl_id, hid_t es_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Aexists}
*/
H5_DLL herr_t H5Aexists_async(const char *app_file, const char *app_func, unsigned app_line, hid_t obj_id,
const char *attr_name, hbool_t *exists, hid_t es_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Aexists_by_name}
*/
H5_DLL herr_t H5Aexists_by_name_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *obj_name, const char *attr_name,
hbool_t *exists, hid_t lapl_id, hid_t es_id);
@@ -1038,6 +1092,11 @@ H5_DLL herr_t H5Aexists_by_name_async(const char *app_file, const char *app_func
*/
H5_DLL herr_t H5Arename_by_name(hid_t loc_id, const char *obj_name, const char *old_attr_name,
const char *new_attr_name, hid_t lapl_id);
/*--------------------------------------------------------------------------*/
/**
* \ingroup ASYNC
* \async_variant_of{H5Aclose}
*/
H5_DLL herr_t H5Aclose_async(const char *app_file, const char *app_func, unsigned app_line, hid_t attr_id,
hid_t es_id);
@@ -1085,9 +1144,28 @@ H5_DLL herr_t H5Aclose_async(const char *app_file, const char *app_func, unsigne
/* Typedefs */
/* Typedef for H5Aiterate1() callbacks */
//! <!-- [H5A_operator1_t_snip] -->
/**
* \brief Typedef for H5Aiterate1() callbacks
*
* \param[in] location_id The identifier for the group, dataset
* or named datatype being iterated over
* \param[in] attr_name The name of the current object attribute
* \param[in,out] operator_data A pointer to the operator data passed in to
* H5Aiterate1()
* \returns The return values from an operator are:
* \li Zero causes the iterator to continue, returning zero when
* all attributes have been processed.
* \li Positive causes the iterator to immediately return that
* positive value, indicating short-circuit success. The
* iterator can be restarted at the next attribute.
* \li Negative causes the iterator to immediately return that value,
* indicating failure. The iterator can be restarted at the next
* attribute.
*/
typedef herr_t (*H5A_operator1_t)(hid_t location_id /*in*/, const char *attr_name /*in*/,
void *operator_data /*in,out*/);
//! <!-- [H5A_operator1_t_snip] -->
/* Function prototypes */
/* --------------------------------------------------------------------------*/
@@ -1163,8 +1241,6 @@ H5_DLL int H5Aget_num_attrs(hid_t loc_id);
*
* \brief Calls a user’s function for each attribute on an object
*
* \todo make prototype parameter match function (idx vs attr_num)
*
* \loc_id
* \param[in,out] idx Starting (in) and ending (out) attribute index
* \param[in] op User's function to pass each attribute to
@@ -1186,29 +1262,12 @@ H5_DLL int H5Aget_num_attrs(hid_t loc_id);
* \p op, is returned in \p idx. If \p idx is the null pointer,
* then all attributes are processed.
*
* The prototype for #H5A_operator1_t is a user defined function
* where:
* The operation receives the identifier for the group, dataset
* or named datatype being iterated over, \p loc_id, the name of
* the current object attribute, \p attr_name, and the pointer to
* the operator data passed in to H5Aiterate1(), \p op_data.
*
* The return values from an operator are:
*
* \li Zero causes the iterator to continue, returning zero when
* all attributes have been processed.
* \li Positive causes the iterator to immediately return that
* positive value, indicating short-circuit success. The
* iterator can be restarted at the next attribute.
* \li Negative causes the iterator to immediately return that value,
* indicating failure. The iterator can be restarted at the next
* attribute.
*
* \todo Add snippet to show H5A_operator1_t.
* \p op is a user-defined function whose prototype is defined as follows:
* \snippet this H5A_operator1_t_snip
* \click4more
*
* \version 1.8.0 The function \p H5Aiterate was renamed to H5Aiterate1()
* and deprecated in this release.
*
* \since 1.0.0
*
*/
src/H5Cpublic.h
+21 -2
View File
@@ -31,15 +31,34 @@
extern "C" {
#endif
enum H5C_cache_incr_mode { H5C_incr__off, H5C_incr__threshold };
enum H5C_cache_incr_mode {
H5C_incr__off,
/**<Automatic cache size increase is disabled, and the remaining increment fields are ignored.*/
enum H5C_cache_flash_incr_mode { H5C_flash_incr__off, H5C_flash_incr__add_space };
H5C_incr__threshold
/**<Automatic cache size increase is enabled using the hit rate threshold algorithm.*/
};
enum H5C_cache_flash_incr_mode {
H5C_flash_incr__off,
/**<Flash cache size increase is disabled.*/
H5C_flash_incr__add_space
/**<Flash cache size increase is enabled using the add space algorithm.*/
};
enum H5C_cache_decr_mode {
H5C_decr__off,
/**<Automatic cache size decrease is disabled.*/
H5C_decr__threshold,
/**<Automatic cache size decrease is enabled using the hit rate threshold algorithm.*/
H5C_decr__age_out,
/**<Automatic cache size decrease is enabled using the ageout algorithm. */
H5C_decr__age_out_with_threshold
/**<Automatic cache size decrease is enabled using the ageout with hit rate threshold algorithm.*/
};
#ifdef __cplusplus
src/H5Dmodule.h
+30 -8
View File
@@ -29,15 +29,37 @@
#define H5_MY_PKG_ERR H5E_DATASET
#define H5_MY_PKG_INIT YES
/**
* \defgroup H5D H5D
* \brief Group Interface
* \details The HDF5 Dataset Interface, H5D, provides a mechanism for managing
* HDF5 datasets, including the transfer of data between memory and
* disk and the description of dataset properties.
/**\defgroup H5D H5D
*
* Use the functions in this module to manage HDF5 datasets, including the
* transfer of data between memory and disk and the description of dataset
* properties. Datasets are used by other HDF5 APIs and referenced either by
* name or by a handle. Such handles can be obtained by either creating or
* opening the dataset.
*
* Typical stages in the HDF5 dataset life cycle are shown below in introductory
* examples.
*
* <table>
* <tr><th>Create</th><th>Read</th></tr>
* <tr valign="top">
* <td>
* \snippet H5D_examples.c create
* </td>
* <td>
* \snippet H5D_examples.c read
* </td>
* <tr><th>Update</th><th>Delete</th></tr>
* <tr valign="top">
* <td>
* \snippet H5D_examples.c update
* </td>
* <td>
* \snippet H5D_examples.c delete
* </td>
* </tr>
* </table>
*
* A Dataset is used by other HDF5 APIs, either by name or by a handle,
* which is obtained by either creating or opening the dataset.
*/
#endif /* H5Dmodule_H */
src/H5Dpublic.h
+263 -152
View File
@@ -39,30 +39,41 @@
/* Public Typedefs */
/*******************/
/* Values for the H5D_LAYOUT property */
//! <!-- [H5D_layout_t_snip] -->
/**
* Values for the H5D_LAYOUT property
*/
typedef enum H5D_layout_t {
H5D_LAYOUT_ERROR = -1,
H5D_COMPACT = 0, /*raw data is very small */
H5D_CONTIGUOUS = 1, /*the default */
H5D_CHUNKED = 2, /*slow and fancy */
H5D_VIRTUAL = 3, /*actual data is stored in other datasets */
H5D_NLAYOUTS = 4 /*this one must be last! */
H5D_COMPACT = 0, /**< raw data is very small */
H5D_CONTIGUOUS = 1, /**< the default */
H5D_CHUNKED = 2, /**< slow and fancy */
H5D_VIRTUAL = 3, /**< actual data is stored in other datasets */
H5D_NLAYOUTS = 4 /**< this one must be last! */
} H5D_layout_t;
//! <!-- [H5D_layout_t_snip] -->
/* Types of chunk index data structures */
//! <!-- [H5D_chunk_index_t_snip] -->
/**
* Types of chunk index data structures
*/
typedef enum H5D_chunk_index_t {
H5D_CHUNK_IDX_BTREE = 0, /* v1 B-tree index (default) */
H5D_CHUNK_IDX_BTREE = 0, /**< v1 B-tree index (default) */
H5D_CHUNK_IDX_SINGLE =
1, /* Single Chunk index (cur dims[]=max dims[]=chunk dims[]; filtered & non-filtered) */
H5D_CHUNK_IDX_NONE = 2, /* Implicit: No Index (H5D_ALLOC_TIME_EARLY, non-filtered, fixed dims) */
H5D_CHUNK_IDX_FARRAY = 3, /* Fixed array (for 0 unlimited dims) */
H5D_CHUNK_IDX_EARRAY = 4, /* Extensible array (for 1 unlimited dim) */
H5D_CHUNK_IDX_BT2 = 5, /* v2 B-tree index (for >1 unlimited dims) */
H5D_CHUNK_IDX_NTYPES /* This one must be last! */
1, /**< Single Chunk index (cur dims[]=max dims[]=chunk dims[]; filtered & non-filtered) */
H5D_CHUNK_IDX_NONE = 2, /**< Implicit: No Index (#H5D_ALLOC_TIME_EARLY, non-filtered, fixed dims) */
H5D_CHUNK_IDX_FARRAY = 3, /**< Fixed array (for 0 unlimited dims) */
H5D_CHUNK_IDX_EARRAY = 4, /**< Extensible array (for 1 unlimited dim) */
H5D_CHUNK_IDX_BT2 = 5, /**< v2 B-tree index (for >1 unlimited dims) */
H5D_CHUNK_IDX_NTYPES /**< This one must be last! */
} H5D_chunk_index_t;
//! <!-- [H5D_chunk_index_t_snip] -->
/* Values for the space allocation time property */
//! <!-- [H5D_alloc_time_t_snip] -->
/**
* Values for the space allocation time property
*/
typedef enum H5D_alloc_time_t {
H5D_ALLOC_TIME_ERROR = -1,
H5D_ALLOC_TIME_DEFAULT = 0,
@@ -70,57 +81,84 @@ typedef enum H5D_alloc_time_t {
H5D_ALLOC_TIME_LATE = 2,
H5D_ALLOC_TIME_INCR = 3
} H5D_alloc_time_t;
//! <!-- [H5D_alloc_time_t_snip] -->
/* Values for the status of space allocation */
//! <!-- [H5D_space_status_t_snip] -->
/**
* Values for the status of space allocation
*/
typedef enum H5D_space_status_t {
H5D_SPACE_STATUS_ERROR = -1,
H5D_SPACE_STATUS_NOT_ALLOCATED = 0,
H5D_SPACE_STATUS_PART_ALLOCATED = 1,
H5D_SPACE_STATUS_ALLOCATED = 2
} H5D_space_status_t;
//! <!-- [H5D_space_status_t_snip] -->
/* Values for time of writing fill value property */
//! <!-- [H5D_fill_time_t_snip] -->
/**
* Values for time of writing fill value property
*/
typedef enum H5D_fill_time_t {
H5D_FILL_TIME_ERROR = -1,
H5D_FILL_TIME_ALLOC = 0,
H5D_FILL_TIME_NEVER = 1,
H5D_FILL_TIME_IFSET = 2
} H5D_fill_time_t;
//! <!-- [H5D_fill_time_t_snip] -->
/* Values for fill value status */
//! <!-- [H5D_fill_value_t_snip] -->
/**
* Values for fill value status
*/
typedef enum H5D_fill_value_t {
H5D_FILL_VALUE_ERROR = -1,
H5D_FILL_VALUE_UNDEFINED = 0,
H5D_FILL_VALUE_DEFAULT = 1,
H5D_FILL_VALUE_USER_DEFINED = 2
} H5D_fill_value_t;
//! <!-- [H5D_fill_value_t_snip] -->
/* Values for VDS bounds option */
//! <!-- [H5D_vds_view_t_snip] -->
/**
* Values for VDS bounds option
*/
typedef enum H5D_vds_view_t {
H5D_VDS_ERROR = -1,
H5D_VDS_FIRST_MISSING = 0,
H5D_VDS_LAST_AVAILABLE = 1
} H5D_vds_view_t;
//! <!-- [H5D_vds_view_t_snip] -->
/* Callback for H5Pset_append_flush() in a dataset access property list */
//! <!-- [H5D_append_cb_t_snip] -->
/**
* Callback for H5Pset_append_flush() in a dataset access property list
*/
typedef herr_t (*H5D_append_cb_t)(hid_t dataset_id, hsize_t *cur_dims, void *op_data);
//! <!-- [H5D_append_cb_t_snip] -->
/** Define the operator function pointer for H5Diterate() */
//! [H5D_operator_t_snip]
//! <!-- [H5D_operator_t_snip] -->
/**
* Define the operator function pointer for H5Diterate()
*/
typedef herr_t (*H5D_operator_t)(void *elem, hid_t type_id, unsigned ndim, const hsize_t *point,
void *operator_data);
//! [H5D_operator_t_snip]
//! <!-- [H5D_operator_t_snip] -->
/** Define the operator function pointer for H5Dscatter() */
//! [H5D_scatter_func_t_snip]
//! <!-- [H5D_scatter_func_t_snip] -->
/**
* Define the operator function pointer for H5Dscatter()
*/
typedef herr_t (*H5D_scatter_func_t)(const void **src_buf /*out*/, size_t *src_buf_bytes_used /*out*/,
void *op_data);
//! [H5D_scatter_func_t_snip]
//! <!-- [H5D_scatter_func_t_snip] -->
/** Define the operator function pointer for H5Dgather() */
//! [H5D_gather_func_t_snip]
//! <!-- [H5D_gather_func_t_snip] -->
/**
* Define the operator function pointer for H5Dgather()
*/
typedef herr_t (*H5D_gather_func_t)(const void *dst_buf, size_t dst_buf_bytes_used, void *op_data);
//! [H5D_gather_func_t_snip]
//! <!-- [H5D_gather_func_t_snip] -->
/********************/
/* Public Variables */
@@ -203,26 +241,8 @@ H5_DLL hid_t H5Dcreate2(hid_t loc_id, const char *name, hid_t type_id, hid_t spa
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Asynchronous version of H5Dcreate2()
*
* \app_file
* \app_func
* \app_line
* \fgdta_loc_id
* \param[in] name Name of the dataset to create
* \type_id
* \space_id
* \lcpl_id
* \dcpl_id
* \dapl_id
* \es_id
*
* \return \hid_t{dataset}
*
* \see H5Dcreate2()
*
* \ingroup ASYNC
* \async_variant_of{H5Dcreate}
*/
H5_DLL hid_t H5Dcreate_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t type_id, hid_t space_id, hid_t lcpl_id, hid_t dcpl_id,
@@ -325,22 +345,8 @@ H5_DLL hid_t H5Dopen2(hid_t loc_id, const char *name, hid_t dapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Asynchronous version of H5Dopen2()
*
* \app_file
* \app_func
* \app_line
* \fgdta_loc_id
* \param[in] name Name of the dataset to open
* \dapl_id
* \es_id
*
* \return \hid_t{dataset}
*
* \see H5Dopen2()
*
* \ingroup ASYNC
* \async_variant_of{H5Dopen}
*/
H5_DLL hid_t H5Dopen_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t dapl_id, hid_t es_id);
@@ -370,24 +376,17 @@ H5_DLL hid_t H5Dget_space(hid_t dset_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Asynchronous version of H5Dget_space()
*
* \app_file
* \app_func
* \app_line
* \dset_id
* \es_id
*
* \return \hid_t{dataspace}
*
* \see H5Dget_space()
*
* \ingroup ASYNC
* \async_variant_of{H5Dget_space}
*/
H5_DLL hid_t H5Dget_space_async(const char *app_file, const char *app_func, unsigned app_line, hid_t dset_id,
hid_t es_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
* \todo Document this function!
*/
H5_DLL herr_t H5Dget_space_status(hid_t dset_id, H5D_space_status_t *allocation);
/**
@@ -614,7 +613,7 @@ H5_DLL herr_t H5Dget_num_chunks(hid_t dset_id, hid_t fspace_id, hsize_t *nchunks
* using the coordinates specified by \p offset.
*
* If the queried chunk does not exist in the file, \p size will
* be set to 0, \p addr to #HADDR_UNDEF, and the buffer \p
* be set to 0, \p addr to \c HADDR_UNDEF, and the buffer \p
* filter_mask will not be modified.
*
* \p offset is a pointer to a one-dimensional array with a size
@@ -649,7 +648,7 @@ H5_DLL herr_t H5Dget_chunk_info_by_coord(hid_t dset_id, const hsize_t *offset, u
* specified by the index index. The chunk belongs to a set of
* chunks in the selection specified by fspace_id. If the queried
* chunk does not exist in the file, the size will be set to 0 and
* address to #HADDR_UNDEF. The value pointed to by filter_mask will
* address to \c HADDR_UNDEF. The value pointed to by filter_mask will
* not be modified. NULL can be passed in for any \p out parameters.
*
* \p chk_idx is the chunk index in the selection. Index value
@@ -684,7 +683,7 @@ H5_DLL herr_t H5Dget_chunk_info(hid_t dset_id, hid_t fspace_id, hsize_t chk_idx,
*
* \dset_id
*
* \return Returns the offset in bytes; otherwise, returns #HADDR_UNDEF,
* \return Returns the offset in bytes; otherwise, returns \c HADDR_UNDEF,
* a negative value.
*
* \details H5Dget_offset() returns the address in the file of
@@ -795,25 +794,8 @@ H5_DLL herr_t H5Dread(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Asynchronous version of H5Dread()
*
* \app_file
* \app_func
* \app_line
* \dset_id Identifier of the dataset to read from
* \param[in] mem_type_id Identifier of the memory datatype
* \param[in] mem_space_id Identifier of the memory dataspace
* \param[in] file_space_id Identifier of the dataset's dataspace in the file
* \param[in] dxpl_id Identifier of a transfer property list
* \param[out] buf Buffer to receive data read from file
* \es_id
*
* \return \herr_t
*
* \see H5Dread()
*
* \ingroup ASYNC
* \async_variant_of{H5Dread}
*/
H5_DLL herr_t H5Dread_async(const char *app_file, const char *app_func, unsigned app_line, hid_t dset_id,
hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id, hid_t dxpl_id,
@@ -932,25 +914,8 @@ H5_DLL herr_t H5Dwrite(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Asynchronous version of H5Dwrite()
*
* \app_file
* \app_func
* \app_line
* \param[in] dset_id Identifier of the dataset to read from
* \param[in] mem_type_id Identifier of the memory datatype
* \param[in] mem_space_id Identifier of the memory dataspace
* \param[in] file_space_id Identifier of the dataset's dataspace in the file
* \dxpl_id
* \param[out] buf Buffer with data to be written to the file
* \es_id
*
* \return \herr_t
*
* \see H5Dwrite()
*
* \ingroup ASYNC
* \async_variant_of{H5Dwrite}
*/
H5_DLL herr_t H5Dwrite_async(const char *app_file, const char *app_func, unsigned app_line, hid_t dset_id,
hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id, hid_t dxpl_id,
@@ -1292,22 +1257,8 @@ H5_DLL herr_t H5Dset_extent(hid_t dset_id, const hsize_t size[]);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Asynchronous version of H5Dset_extent()
*
* \app_file
* \app_func
* \app_line
* \dset_id
* \param[in] size[] Array containing the new magnitude of each dimension
* of the dataset
* \es_id
*
* \return \herr_t
*
* \see H5Dset_extent()
*
* \ingroup ASYNC
* \async_variant_of{H5Dset_extent}
*/
H5_DLL herr_t H5Dset_extent_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t dset_id, const hsize_t size[], hid_t es_id);
@@ -1536,20 +1487,8 @@ H5_DLL herr_t H5Dclose(hid_t dset_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Asynchronous version of H5Dclose()
*
* \app_file
* \app_func
* \app_line
* \dset_id
* \es_id
*
* \return \herr_t
*
* \see H5Dclose()
*
* \ingroup ASYNC
* \async_variant_of{H5Dclose}
*/
H5_DLL herr_t H5Dclose_async(const char *app_file, const char *app_func, unsigned app_line, hid_t dset_id,
hid_t es_id);
@@ -1608,9 +1547,181 @@ H5_DLL herr_t H5Dget_chunk_index_type(hid_t did, H5D_chunk_index_t *idx_type);
/* Typedefs */
/* Function prototypes */
H5_DLL hid_t H5Dcreate1(hid_t loc_id, const char *name, hid_t type_id, hid_t space_id, hid_t dcpl_id);
H5_DLL hid_t H5Dopen1(hid_t loc_id, const char *name);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Creates a dataset at the specified location
*
* \fgdta_loc_id
* \param[in] name Name of the dataset to create
* \type_id
* \space_id
* \dcpl_id
*
* \return \hid_t{dataset}
*
* \deprecated This function is deprecated in favor of the function H5Dcreate2()
* or the macro H5Dcreate().
*
* \details H5Dcreate1() creates a data set with a name, \p name, in the
* location specified by the identifier \p loc_id. \p loc_id may be a
* file, group, dataset, named datatype or attribute. If an attribute,
* dataset, or named datatype is specified for \p loc_id then the
* dataset will be created at the location where the attribute,
* dataset, or named datatype is attached.
*
* \p name can be a relative path based at \p loc_id or an absolute
* path from the root of the file. Use of this function requires that
* any intermediate groups specified in the path already exist.
*
* The dataset’s datatype and dataspace are specified by \p type_id and
* \p space_id, respectively. These are the datatype and dataspace of
* the dataset as it will exist in the file, which may differ from the
* datatype and dataspace in application memory.
*
* Names within a group are unique: H5Dcreate1() will return an error
* if a link with the name specified in name already exists at the
* location specified in \p loc_id.
*
* As is the case for any object in a group, the length of a dataset
* name is not limited.
*
* \p dcpl_id is an #H5P_DATASET_CREATE property list created with \p
* H5reate1() and initialized with various property list functions
* described in Property List Interface.
*
* H5Dcreate() and H5Dcreate_anon() return an error if the dataset’s
* datatype includes a variable-length (VL) datatype and the fill value
* is undefined, i.e., set to \c NULL in the dataset creation property
* list. Such a VL datatype may be directly included, indirectly
* included as part of a compound or array datatype, or indirectly
* included as part of a nested compound or array datatype.
*
* H5Dcreate() and H5Dcreate_anon() return a dataset identifier for
* success or a negative value for failure. The dataset identifier
* should eventually be closed by calling H5Dclose() to release
* resources it uses.
*
* See H5Dcreate_anon() for discussion of the differences between
* H5Dcreate() and H5Dcreate_anon().
*
* The HDF5 library provides flexible means of specifying a fill value,
* of specifying when space will be allocated for a dataset, and of
* specifying when fill values will be written to a dataset.
*
* \version 1.8.0 Function H5Dcreate() renamed to H5Dcreate1() and deprecated in this release.
* \since 1.0.0
*
* \see H5Dopen2(), H5Dclose(), H5Tset_size()
*
*/
H5_DLL hid_t H5Dcreate1(hid_t loc_id, const char *name, hid_t type_id, hid_t space_id, hid_t dcpl_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Opens an existing dataset
*
* \fgdta_loc_id
* \param[in] name Name of the dataset to access
*
* \return \hid_t{dataset}
*
* \deprecated This function is deprecated in favor of the function H5Dopen2()
* or the macro H5Dopen().
*
* \details H5Dopen1() opens an existing dataset for access at the location
* specified by \p loc_id. \p loc_id may be a file, group, dataset,
* named datatype or attribute. If an attribute, dataset, or named
* datatype is specified for loc_id then the dataset will be opened at
* the location where the attribute, dataset, or named datatype is
* attached. name is a dataset name and is used to identify the dataset
* in the file.
*
* A dataset opened with this function should be closed with H5Dclose()
* when the dataset is no longer needed so that resource leaks will not
* develop.
*
* \version 1.8.0 Function H5Dopen() renamed to H5Dopen1() and deprecated in this release.
* \since 1.0.0
*
*/
H5_DLL hid_t H5Dopen1(hid_t loc_id, const char *name);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Extends a dataset
*
* \dset_id
* \param[in] size Array containing the new size of each dimension
*
* \return \herr_t
*
* \deprecated This function is deprecated in favor of the function H5Dset_extent().
*
* \details H5Dextend() verifies that the dataset is at least of size \p size,
* extending it if necessary. The dimensionality of size is the same as
* that of the dataspace of the dataset being changed.
*
* This function can be applied to the following datasets:
* \li Any dataset with unlimited dimensions
* \li A dataset with fixed dimensions if the current dimension sizes
* are less than the maximum sizes set with \c maxdims
* (see H5Screate_simple())
*
* Space on disk is immediately allocated for the new dataset extent if
* the dataset’s space allocation time is set to
* #H5D_ALLOC_TIME_EARLY. Fill values will be written to the dataset if
* the dataset’s fill time is set to #H5D_FILL_TIME_IFSET or
* #H5D_FILL_TIME_ALLOC. (See H5Pset_fill_time() and
* H5Pset_alloc_time().)
*
* This function ensures that the dataset dimensions are of at least
* the sizes specified in size. The function H5Dset_extent() must be
* used if the dataset dimension sizes are are to be reduced.
*
* \version 1.8.0 Function Function deprecated in this release. Parameter size
* syntax changed to \Code{const hsize_t size[]} in this release.
*
*/
H5_DLL herr_t H5Dextend(hid_t dset_id, const hsize_t size[]);
/**
* --------------------------------------------------------------------------
* \ingroup H5D
*
* \brief Reclaims variable-length (VL) datatype memory buffers
*
* \type_id
* \space_id
* \dxpl_id
* \param[in] buf Pointer to the buffer to be reclaimed
*
* \return \herr_t
*
* \deprecated This function has been deprecated in HDF5-1.12 in favor of the
* function H5Treclaim().
*
* \details H5Dvlen_reclaim() reclaims memory buffers created to store VL
* datatypes.
*
* The \p type_id must be the datatype stored in the buffer. The \p
* space_id describes the selection for the memory buffer to free the
* VL datatypes within. The \p dxpl_id is the dataset transfer property
* list which was used for the I/O transfer to create the buffer. And
* \p buf is the pointer to the buffer to be reclaimed.
*
* The VL structures (\ref hvl_t) in the user's buffer are modified to
* zero out the VL information after the memory has been reclaimed.
*
* If nested VL datatypes were used to create the buffer, this routine
* frees them from the bottom up, releasing all the memory without
* creating memory leaks.
*
* \version 1.12.0 Routine was deprecated
*
*/
H5_DLL herr_t H5Dvlen_reclaim(hid_t type_id, hid_t space_id, hid_t dxpl_id, void *buf);
#endif /* H5_NO_DEPRECATED_SYMBOLS */
src/H5ESmodule.h
+33
View File
@@ -29,4 +29,37 @@
#define H5_MY_PKG_ERR H5E_EVENTSET
#define H5_MY_PKG_INIT YES
/**
* \defgroup H5ES H5ES
* \brief Event Set Interface
*
* \details \Bold{This interface can be only used with the HDF5 VOL connectors that
* enable the asynchronous feature in HDF5.} The native HDF5 library has
* only synchronous operations.
*
* HDF5 VOL connectors with support for asynchronous operations:
* - ASYNC
* - DAOS
*
* \par Example:
* \code
* fid = H5Fopen(..);
* gid = H5Gopen(fid, ..); //Starts when H5Fopen completes
* did = H5Dopen(gid, ..); //Starts when H5Gopen completes
*
* es_id = H5EScreate(); // Create event set for tracking async operations
* status = H5Dwrite_async(did, .., es_id); //Asynchronous, starts when H5Dopen completes,
* // may run concurrently with other H5Dwrite_async
* // in event set.
* status = H5Dwrite_async(did, .., es_id); //Asynchronous, starts when H5Dopen completes,
* // may run concurrently with other H5Dwrite_async
* // in event set....
* <other user code>
* ...
* H5ESwait(es_id); // Wait for operations in event set to complete, buffers
* // used for H5Dwrite_async must only be changed after wait
* // returns.
* \endcode
*/
#endif /* H5ESmodule_H */
src/H5ESpublic.h
+157 -10
View File
@@ -47,20 +47,24 @@ typedef enum H5ES_status_t {
H5ES_STATUS_FAIL /* An operation has completed, but failed */
} H5ES_status_t;
/* Information about failed operations in event set */
//! <!-- [H5ES_err_info_t_snip] -->
/**
* Information about failed operations in event set
*/
typedef struct H5ES_err_info_t {
/* Operation info */
char * api_name; /* Name of HDF5 API routine called */
char * api_args; /* "Argument string" for arguments to HDF5 API routine called */
char * app_file_name; /* Name of source file where the HDF5 API routine was called */
char * app_func_name; /* Name of function where the HDF5 API routine was called */
unsigned app_line_num; /* Line # of source file where the HDF5 API routine was called */
uint64_t op_ins_count; /* Counter of operation's insertion into event set */
uint64_t op_ins_ts; /* Timestamp for when the operation was inserted into the event set */
char * api_name; /**< Name of HDF5 API routine called */
char * api_args; /**< "Argument string" for arguments to HDF5 API routine called */
char * app_file_name; /**< Name of source file where the HDF5 API routine was called */
char * app_func_name; /**< Name of function where the HDF5 API routine was called */
unsigned app_line_num; /**< Line # of source file where the HDF5 API routine was called */
uint64_t op_ins_count; /**< Counter of operation's insertion into event set */
uint64_t op_ins_ts; /**< Timestamp for when the operation was inserted into the event set */
/* Error info */
hid_t err_stack_id; /* ID for error stack from failed operation */
hid_t err_stack_id; /**< ID for error stack from failed operation */
} H5ES_err_info_t;
//! <!-- [H5ES_err_info_t_snip] -->
/*
H5ES_op_info_t:
@@ -119,14 +123,157 @@ How to Trace Async Operations?
extern "C" {
#endif
H5_DLL hid_t H5EScreate(void);
/**
* \ingroup H5ES
*
* \brief Creates an event set
*
* \returns \hid_ti{event set}
*
* \details H5EScreate() creates a new event set and returns a corresponding
* event set identifier.
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5EScreate(void);
/**
* \ingroup H5ES
*
* \brief Waits for operations in event set to complete
*
* \es_id
* \param[in] timeout Total time in nanoseconds to wait for all operations in
* the event set to complete
* \param[out] num_in_progress The number of operations still in progress
* \param[out] err_occurred Flag if an operation in the event set failed
* \returns \herr_t
*
* \details H5ESwait() waits for operations in an event set \p es_id to wait
* with \p timeout.
*
* Timeout value is in nanoseconds, and is for the H5ESwait() call and
* not for each individual operation in the event set. For example, if
* "10" is passed as a timeout value and the event set waited 4
* nanoseconds for the first operation to complete, the remaining
* operations would be allowed to wait for at most 6 nanoseconds more,
* i.e., the timeout value used across all operations in the event set
* until it reaches 0, then any remaining operations are only checked
* for completion, not waited on.
*
* This call will stop waiting on operations and will return
* immediately if an operation fails. If a failure occurs, the value
* returned for the number of operations in progress may be inaccurate.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5ESwait(hid_t es_id, uint64_t timeout, size_t *num_in_progress, hbool_t *err_occurred);
/**
* \ingroup H5ES
*
* \brief Retrieves number of events in an event set
*
* \es_id
* \param[out] count The number of events in the event set
* \returns \herr_t
*
* \details H5ESget_count() retrieves number of events in an event set specified
* by \p es_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5ESget_count(hid_t es_id, size_t *count);
/**
* \ingroup H5ES
*
* \todo Fill in the blanks!
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5ESget_op_counter(hid_t es_id, uint64_t *counter);
/**
* \ingroup H5ES
*
* \brief Checks for failed operations
*
* \es_id
* \param[out] err_occurred Status indicating if error is present in the event
* set
* \returns \herr_t
*
* \details H5ESget_err_status() checks if event set specified by es_id has
* failed operations.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5ESget_err_status(hid_t es_id, hbool_t *err_occurred);
/**
* \ingroup H5ES
*
* \brief Retrieves the number of failed operations
*
* \es_id
* \param[out] num_errs Number of errors
* \returns \herr_t
*
* \details H5ESget_err_count() retrieves the number of failed operations in an
* event set specified by \p es_id.
*
* The function does not wait for active operations to complete, so
* count may not include all failures.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5ESget_err_count(hid_t es_id, size_t *num_errs);
/**
* \ingroup H5ES
*
* \brief Retrieves information about failed operations
*
* \es_id
* \param[in] num_err_info The number of elements in \p err_info array
* \param[out] err_info Array of structures
* \param[out] err_cleared Number of cleared errors
* \returns \herr_t
*
* \details H5ESget_err_info() retrieves information about failed operations in
* an event set specified by \p es_id. The strings retrieved for each
* error info must be released by calling H5free_memory().
*
* Below is the description of the \ref H5ES_err_info_t structure:
* \snippet this H5ES_err_info_t_snip
* \click4more
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5ESget_err_info(hid_t es_id, size_t num_err_info, H5ES_err_info_t err_info[],
size_t *err_cleared);
/**
* \ingroup H5ES
*
* \brief Terminates access to an event set
*
* \es_id
* \returns \herr_t
*
* \details H5ESclose() terminates access to an event set specified by \p es_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5ESclose(hid_t es_id);
#ifdef __cplusplus
src/H5Emodule.h
+29
View File
@@ -29,4 +29,33 @@
#define H5_MY_PKG_ERR H5E_ERROR
#define H5_MY_PKG_INIT YES
/**
* \defgroup H5E H5E
* \brief Error Handling Interface
*
* \details The Error interface provides error handling in the form of a stack.
* The \Code{FUNC_ENTER} macro clears the error stack whenever an
* interface function is entered. When an error is detected, an entry
* is pushed onto the stack. As the functions unwind, additional
* entries are pushed onto the stack. The API function will return some
* indication that an error occurred and the application can print the
* error stack.
*
* Certain API functions in the \c H5E package, such as H5Eprint1(), do
* not clear the error stack. Otherwise, any function which does not
* have an underscore immediately after the package name will clear the
* error stack. For instance, H5Fopen() clears the error stack while
* \Code{H5F_open} does not.
*
* An error stack has a fixed maximum size. If this size is exceeded
* then the stack will be truncated and only the inner-most functions
* will have entries on the stack. This is expected to be a rare
* condition.
*
* Each thread has its own error stack, but since multi-threading has
* not been added to the library yet, this package maintains a single
* error stack. The error stack is statically allocated to reduce the
* complexity of handling errors within the \c H5E package.
*/
#endif /* H5Emodule_H */
src/H5Epublic.h
+749 -39
View File
@@ -26,18 +26,29 @@
/* Value for the default error stack */
#define H5E_DEFAULT (hid_t)0
/* Different kinds of error information */
/**
* Different kinds of error information
*/
typedef enum H5E_type_t { H5E_MAJOR, H5E_MINOR } H5E_type_t;
/* Information about an error; element of error stack */
/**
* Information about an error; element of error stack
*/
typedef struct H5E_error2_t {
hid_t cls_id; /*class ID */
hid_t maj_num; /*major error ID */
hid_t min_num; /*minor error number */
unsigned line; /*line in file where error occurs */
const char *func_name; /*function in which error occurred */
const char *file_name; /*file in which error occurred */
const char *desc; /*optional supplied description */
hid_t cls_id;
/**< Class ID */
hid_t maj_num;
/**< Major error ID */
hid_t min_num;
/**< Minor error number */
unsigned line;
/**< Line in file where error occurs */
const char *func_name;
/**< Function in which error occurred */
const char *file_name;
/**< File in which error occurred */
const char *desc;
/**< Optional supplied description */
} H5E_error2_t;
/* When this header is included from a private header, don't make calls to H5open() */
@@ -138,10 +149,12 @@ H5_DLLVAR hid_t H5E_ERR_CLS_g;
goto label; \
}
/* Error stack traversal direction */
/**
* Error stack traversal direction
*/
typedef enum H5E_direction_t {
H5E_WALK_UPWARD = 0, /*begin deep, end at API function */
H5E_WALK_DOWNWARD = 1 /*begin at API function, end deep */
H5E_WALK_UPWARD = 0, /**< begin w/ most specific error, end at API function */
H5E_WALK_DOWNWARD = 1 /**< begin at API function, end w/ most specific error */
} H5E_direction_t;
#ifdef __cplusplus
@@ -149,30 +162,498 @@ extern "C" {
#endif
/* Error stack traversal callback function pointers */
//! <!-- [H5E_walk2_t_snip] -->
/**
* \brief Callback function for H5Ewalk2()
*
* \param[in] n Indexed error position in the stack
* \param[in] err_desc Pointer to a data structure describing the error
* \param[in] client_data Pointer to client data in the format expected by the
* user-defined function
* \return \herr_t
*/
typedef herr_t (*H5E_walk2_t)(unsigned n, const H5E_error2_t *err_desc, void *client_data);
//! <!-- [H5E_walk2_t_snip] -->
//! <!-- [H5E_auto2_t_snip] -->
/**
* \brief Callback function for H5Eset_auto2()
*
* \estack_id{estack}
* \param[in] client_data Pointer to client data in the format expected by the
* user-defined function
* \return \herr_t
*/
typedef herr_t (*H5E_auto2_t)(hid_t estack, void *client_data);
//! <!-- [H5E_auto2_t_snip] -->
/* Public API functions */
H5_DLL hid_t H5Eregister_class(const char *cls_name, const char *lib_name, const char *version);
H5_DLL herr_t H5Eunregister_class(hid_t class_id);
H5_DLL herr_t H5Eclose_msg(hid_t err_id);
H5_DLL hid_t H5Ecreate_msg(hid_t cls, H5E_type_t msg_type, const char *msg);
H5_DLL hid_t H5Ecreate_stack(void);
H5_DLL hid_t H5Eget_current_stack(void);
H5_DLL herr_t H5Eappend_stack(hid_t dst_stack_id, hid_t src_stack_id, hbool_t close_source_stack);
H5_DLL herr_t H5Eclose_stack(hid_t stack_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Registers a client library or application program to the HDF5 error API
*
* \param[in] cls_name Name of the error class
* \param[in] lib_name Name of the client library or application to which the error class belongs
* \param[in] version Version of the client library or application to which the
error class belongs. Can be \c NULL.
* \return Returns a class identifier on success; otherwise returns H5I_INVALID_ID.
*
* \details H5Eregister_class() registers a client library or application
* program to the HDF5 error API so that the client library or
* application program can report errors together with the HDF5
* library. It receives an identifier for this error class for further
* error operations. The library name and version number will be
* printed out in the error message as a preamble.
*
* \since 1.8.0
*/
H5_DLL hid_t H5Eregister_class(const char *cls_name, const char *lib_name, const char *version);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Removes an error class
*
* \param[in] class_id Error class identifier.
* \return \herr_t
*
* \details H5Eunregister_class() removes the error class specified by \p
* class_id. All the major and minor errors in this class will also be
* closed.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eunregister_class(hid_t class_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Closes an error message
*
* \param[in] err_id An error message identifier
* \return \herr_t
*
* \details H5Eclose_msg() closes an error message identifier, which can be
* either a major or minor message.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eclose_msg(hid_t err_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Adds a major error message to an error class
*
* \param[in] cls An error class identifier
* \param[in] msg_type The type of the error message
* \param[in] msg Major error message
* \return \herr_t
*
* \details H5Ecreate_msg() adds an error message to an error class defined by
* client library or application program. The error message can be
* either major or minor as indicated by the parameter \p msg_type.
*
* Use H5Eclose_msg() to close the message identifier returned by this
* function.
*
* \since 1.8.0
*/
H5_DLL hid_t H5Ecreate_msg(hid_t cls, H5E_type_t msg_type, const char *msg);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Creates a new, empty error stack
*
* \return \hid_ti{error stack}
*
* \details H5Ecreate_stack() creates a new empty error stack and returns the
* new stack’s identifier. Use H5Eclose_stack() to close the error stack
* identifier returned by this function.
*
* \since 1.8.0
*/
H5_DLL hid_t H5Ecreate_stack(void);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Returns a copy of the current error stack
*
* \return \hid_ti{error stack}
*
* \details H5Eget_current_stack() copies the current error stack and returns an
* error stack identifier for the new copy.
*
* \since 1.8.0
*/
H5_DLL hid_t H5Eget_current_stack(void);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Appends one error stack to another, optionally closing the source
* stack.
*
* \estack_id{dst_stack_id}
* \estack_id{src_stack_id}
* \param[in] close_source_stack Flag to indicate whether to close the source stack
* \return \herr_t
*
* \details H5Eappend_stack() appends the messages from error stack
* \p src_stack_id to the error stack \p dst_stack_id.
* If \p close_source_stack is \c TRUE, the source error stack
* will be closed.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eappend_stack(hid_t dst_stack_id, hid_t src_stack_id, hbool_t close_source_stack);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Closes an error stack handle
*
* \estack_id{stack_id}
*
* \return \herr_t
*
* \details H5Eclose_stack() closes the error stack handle \p stack_id
* and releases its resources. #H5E_DEFAULT cannot be closed.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eclose_stack(hid_t stack_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Retrieves error class name
*
* \param[in] class_id Error class identifier
* \param[out] name Buffer for the error class name
* \param[in] size The maximum number of characters the class name to be returned
* by this function in\p name.
* \return Returns non-negative value as on success; otherwise returns negative value.
*
* \details H5Eget_class_name() retrieves the name of the error class specified
* by the class identifier. If non-NULL pointer is passed in for \p
* name and \p size is greater than zero, the class name of \p size
* long is returned. The length of the error class name is also
* returned. If NULL is passed in as \p name, only the length of class
* name is returned. If zero is returned, it means no name. The user is
* responsible for allocating sufficient buffer space for the name.
*
* \since 1.8.0
*/
H5_DLL ssize_t H5Eget_class_name(hid_t class_id, char *name, size_t size);
H5_DLL herr_t H5Eset_current_stack(hid_t err_stack_id);
H5_DLL herr_t H5Epush2(hid_t err_stack, const char *file, const char *func, unsigned line, hid_t cls_id,
hid_t maj_id, hid_t min_id, const char *msg, ...);
H5_DLL herr_t H5Epop(hid_t err_stack, size_t count);
H5_DLL herr_t H5Eprint2(hid_t err_stack, FILE *stream);
H5_DLL herr_t H5Ewalk2(hid_t err_stack, H5E_direction_t direction, H5E_walk2_t func, void *client_data);
H5_DLL herr_t H5Eget_auto2(hid_t estack_id, H5E_auto2_t *func, void **client_data);
H5_DLL herr_t H5Eset_auto2(hid_t estack_id, H5E_auto2_t func, void *client_data);
H5_DLL herr_t H5Eclear2(hid_t err_stack);
H5_DLL herr_t H5Eauto_is_v2(hid_t err_stack, unsigned *is_stack);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Replaces the current error stack
*
* \estack_id{err_stack_id}
*
* \return \herr_t
*
* \details H5Eset_current_stack() replaces the content of the current error
* stack with a copy of the content of the error stack specified by
* \p err_stack_id, and it closes the error stack specified by
* \p err_stack_id.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eset_current_stack(hid_t err_stack_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Pushes a new error record onto an error stack
*
* \estack_id{err_stack}. If the identifier is #H5E_DEFAULT, the error record
* will be pushed to the current stack.
* \param[in] file Name of the file in which the error was detected
* \param[in] func Name of the function in which the error was detected
* \param[in] line Line number in the file where the error was detected
* \param[in] cls_id Error class identifier
* \param[in] maj_id Major error identifier
* \param[in] min_id Minor error identifier
* \param[in] msg Error description string
* \return \herr_t
*
* \details H5Epush2() pushes a new error record onto the error stack specified
* by \p err_stack.\n
* The error record contains the error class identifier \p cls_id, the
* major and minor message identifiers \p maj_id and \p min_id, the
* function name \p func where the error was detected, the file name \p
* file and line number \p line in the file where the error was
* detected, and an error description \p msg.\n
* The major and minor errors must be in the same error class.\n
* The function name, filename, and error description strings must be
* statically allocated.\n
* \p msg can be a format control string with additional
* arguments. This design of appending additional arguments is similar
* to the system and C functions printf() and fprintf().
*
* \since 1.8.0
*/
H5_DLL herr_t H5Epush2(hid_t err_stack, const char *file, const char *func, unsigned line, hid_t cls_id,
hid_t maj_id, hid_t min_id, const char *msg, ...);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Deletes specified number of error messages from the error stack
*
* \estack_id{err_stack}
* \param[in] count The number of error messages to be deleted from the top
* of error stack
* \return \herr_t
*
* \details H5Epop() deletes the number of error records specified in \p count
* from the top of the error stack specified by \p err_stack (including
* major, minor messages and description). The number of error messages
* to be deleted is specified by \p count.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Epop(hid_t err_stack, size_t count);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Prints the specified error stack in a default manner
*
* \estack_id{err_stack}
* \param[in] stream File pointer, or \c NULL for \c stderr
* \return \herr_t
*
* \details H5Eprint2() prints the error stack specified by \p err_stack on the
* specified stream, \p stream. Even if the error stack is empty, a
* one-line message of the following form will be printed:
* \code{.unparsed}
* HDF5-DIAG: Error detected in HDF5 library version: 1.5.62 thread 0.
* \endcode
*
* A similar line will appear before the error messages of each error
* class stating the library name, library version number, and thread
* identifier.
*
* If \p err_stack is #H5E_DEFAULT, the current error stack will be
* printed.
*
* H5Eprint2() is a convenience function for H5Ewalk2() with a function
* that prints error messages. Users are encouraged to write their own
* more specific error handlers.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eprint2(hid_t err_stack, FILE *stream);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Walks the specified error stack, calling the specified function
*
* \estack_id{err_stack}
* \param[in] direction Direction in which the error stack is to be walked
* \param[in] func Function to be called for each error encountered
* \param[in] client_data Data to be passed to \p func
* \return \herr_t
*
* \details H5Ewalk2() walks the error stack specified by err_stack for the
* current thread and calls the function specified in \p func for each
* error along the way.
*
* If the value of \p err_stack is #H5E_DEFAULT, then H5Ewalk2() walks
* the current error stack.
*
* \p direction specifies whether the stack is walked from the inside
* out or the outside in. A value of #H5E_WALK_UPWARD means to begin
* with the most specific error and end at the API; a value of
* #H5E_WALK_DOWNWARD means to start at the API and end at the
* innermost function where the error was first detected.
*
* \p func, a function conforming to the #H5E_walk2_t prototype, will
* be called for each error in the error stack. Its arguments will
* include an index number \c n (beginning at zero regardless of stack
* traversal direction), an error stack entry \c err_desc, and the \c
* client_data pointer passed to H5Eprint(). The #H5E_walk2_t prototype
* is as follows:
* \snippet this H5E_walk2_t_snip
*
* \since 1.8.0
*/
H5_DLL herr_t H5Ewalk2(hid_t err_stack, H5E_direction_t direction, H5E_walk2_t func, void *client_data);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Returns the settings for the automatic error stack traversal
* function and its data
*
* \estack_id
* \param[out] func The function currently set to be called upon an error condition
* \param[out] client_data Data currently set to be passed to the error function
* \return \herr_t
*
* \details H5Eget_auto2() returns the settings for the automatic error stack
* traversal function, \p func, and its data, \p client_data, that are
* associated with the error stack specified by \p estack_id.
*
* Either or both of the \p func and \p client_data arguments may be
* \c NULL, in which case the value is not returned.
*
* The library initializes its default error stack traversal functions
* to H5Eprint1() and H5Eprint2(). A call to H5Eget_auto2() returns
* H5Eprint2() or the user-defined function passed in through
* H5Eset_auto2(). A call to H5Eget_auto1() returns H5Eprint1() or the
* user-defined function passed in through H5Eset_auto1(). However, if
* the application passes in a user-defined function through
* H5Eset_auto1(), it should call H5Eget_auto1() to query the traversal
* function. If the application passes in a user-defined function
* through H5Eset_auto2(), it should call H5Eget_auto2() to query the
* traversal function.
*
* Mixing the new style and the old style functions will cause a
* failure. For example, if the application sets a user-defined
* old-style traversal function through H5Eset_auto1(), a call to
* H5Eget_auto2() will fail and will indicate that the application has
* mixed H5Eset_auto1() and H5Eget_auto2(). On the other hand, mixing
* H5Eset_auto2() and H5Eget_auto1() will also cause a failure. But if
* the traversal functions are the library’s default H5Eprint1() or
* H5Eprint2(), mixing H5Eset_auto1() and H5Eget_auto2() or mixing
* H5Eset_auto2() and H5Eget_auto1() does not fail.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eget_auto2(hid_t estack_id, H5E_auto2_t *func, void **client_data);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Turns automatic error printing on or off
*
* \estack_id
* \param[in] func Function to be called upon an error condition
* \param[in] client_data Data passed to the error function
* \return \herr_t
*
* \details H5Eset_auto2() turns on or off automatic printing of errors for the
* error stack specified with \p estack_id. An \p estack_id value of
* #H5E_DEFAULT indicates the current stack.
*
* When automatic printing is turned on, by the use of a non-null \p func
* pointer, any API function which returns an error indication will
* first call \p func, passing it \p client_data as an argument.
*
* \p func, a function compliant with the #H5E_auto2_t prototype, is
* defined in the H5Epublic.h source code file as:
* \snippet this H5E_auto2_t_snip
*
* When the library is first initialized, the auto printing function is
* set to H5Eprint2() (cast appropriately) and \p client_data is the
* standard error stream pointer, \c stderr.
*
* Automatic stack traversal is always in the #H5E_WALK_DOWNWARD
* direction.
*
* Automatic error printing is turned off with a H5Eset_auto2() call
* with a \c NULL \p func pointer.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eset_auto2(hid_t estack_id, H5E_auto2_t func, void *client_data);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Clears the specified error stack or the error stack for the current thread
*
* \estack_id{err_stack}
* \return \herr_t
*
* \details H5Eclear2() clears the error stack specified by \p err_stack, or, if
* \p err_stack is set to #H5E_DEFAULT, the error stack for the current
* thread.
*
* \p err_stack is an error stack identifier, such as that returned by
* H5Eget_current_stack().
*
* The current error stack is also cleared whenever an API function is
* called, with certain exceptions (for instance, H5Eprint1() or
* H5Eprint2()).
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eclear2(hid_t err_stack);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Determines the type of error stack
*
* \estack_id{err_stack}
* \param[out] is_stack A flag indicating which error stack \c typedef the
* specified error stack conforms to
*
* \return \herr_t
*
* \details H5Eauto_is_v2() determines whether the error auto reporting function
* for an error stack conforms to the #H5E_auto2_t \c typedef or the
* #H5E_auto1_t \c typedef.
*
* The \p is_stack parameter is set to 1 if the error stack conforms to
* #H5E_auto2_t and 0 if it conforms to #H5E_auto1_t.
*
* \since 1.8.0
*/
H5_DLL herr_t H5Eauto_is_v2(hid_t err_stack, unsigned *is_stack);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Retrieves an error message
*
* \param[in] msg_id Error message identifier
* \param[out] type The type of the error message Valid values are #H5E_MAJOR
* and #H5E_MINOR.
* \param[out] msg Error message buffer
* \param[in] size The length of error message to be returned by this function
* \return Returns the size of the error message in bytes on success; otherwise
* returns a negative value.
*
* \details H5Eget_msg() retrieves the error message including its length and
* type. The error message is specified by \p msg_id. The user is
* responsible for passing in sufficient buffer space for the
* message. If \p msg is not NULL and \p size is greater than zero, the
* error message of \p size long is returned. The length of the message
* is also returned. If NULL is passed in as \p msg, only the length
* and type of the message is returned. If the return value is zero, it
* means there is no message.
*
* \since 1.8.0
*/
H5_DLL ssize_t H5Eget_msg(hid_t msg_id, H5E_type_t *type, char *msg, size_t size);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Retrieves the number of error messages in an error stack
*
* \estack_id{error_stack_id}
* \return Returns a non-negative value on success; otherwise returns a negative value.
*
* \details H5Eget_num() retrieves the number of error records in the error
* stack specified by \p error_stack_id (including major, minor
* messages and description).
*
* \since 1.8.0
*/
H5_DLL ssize_t H5Eget_num(hid_t error_stack_id);
/* Symbols defined for compatibility with previous versions of the HDF5 API.
@@ -189,30 +670,259 @@ H5_DLL ssize_t H5Eget_num(hid_t error_stack_id);
typedef hid_t H5E_major_t;
typedef hid_t H5E_minor_t;
/* Information about an error element of error stack. */
/**
* Information about an error element of error stack.
*/
typedef struct H5E_error1_t {
H5E_major_t maj_num; /*major error number */
H5E_minor_t min_num; /*minor error number */
const char *func_name; /*function in which error occurred */
const char *file_name; /*file in which error occurred */
unsigned line; /*line in file where error occurs */
const char *desc; /*optional supplied description */
H5E_major_t maj_num; /**< major error number */
H5E_minor_t min_num; /**< minor error number */
const char *func_name; /**< function in which error occurred */
const char *file_name; /**< file in which error occurred */
unsigned line; /**< line in file where error occurs */
const char *desc; /**< optional supplied description */
} H5E_error1_t;
/* Error stack traversal callback function pointers */
//! <!-- [H5E_walk1_t_snip] -->
/**
* \brief Callback function for H5Ewalk1()
*
* \param[in] n Indexed error position in the stack
* \param[in] err_desc Pointer to a data structure describing the error
* \param[in] client_data Pointer to client data in the format expected by the
* user-defined function
* \return \herr_t
*/
typedef herr_t (*H5E_walk1_t)(int n, H5E_error1_t *err_desc, void *client_data);
//! <!-- [H5E_walk1_t_snip] -->
//! <!-- [H5E_auto1_t_snip] -->
/**
* \brief Callback function for H5Eset_auto1()
*
* \param[in] client_data Pointer to client data in the format expected by the
* user-defined function
* \return \herr_t
*/
typedef herr_t (*H5E_auto1_t)(void *client_data);
//! <!-- [H5E_auto1_t_snip] -->
/* Function prototypes */
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Clears the error stack for the current thread
*
* \return \herr_t
*
* \details H5Eclear1() clears the error stack for the current thread.\n
* The stack is also cleared whenever an API function is called, with
* certain exceptions (for instance, H5Eprint1()).
*
* \deprecated 1.8.0 Function H5Eclear() renamed to H5Eclear1() and deprecated
* in this release.
*/
H5_DLL herr_t H5Eclear1(void);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Returns the current settings for the automatic error stack traversal
* function and its data
*
* \param[out] func Current setting for the function to be called upon an error
* condition
* \param[out] client_data Current setting for the data passed to the error
* function
* \return \herr_t
*
* \details H5Eget_auto1() returns the current settings for the automatic error
* stack traversal function, \p func, and its data,
* \p client_data. Either or both arguments may be \c NULL, in which case the
* value is not returned.
*
* The library initializes its default error stack traversal functions
* to H5Eprint1() and H5Eprint2(). A call to H5Eget_auto2() returns
* H5Eprint2() or the user-defined function passed in through
* H5Eset_auto2(). A call to H5Eget_auto1() returns H5Eprint1() or the
* user-defined function passed in through H5Eset_auto1(). However, if
* the application passes in a user-defined function through
* H5Eset_auto1(), it should call H5Eget_auto1() to query the traversal
* function. If the application passes in a user-defined function
* through H5Eset_auto2(), it should call H5Eget_auto2() to query the
* traversal function.
*
* Mixing the new style and the old style functions will cause a
* failure. For example, if the application sets a user-defined
* old-style traversal function through H5Eset_auto1(), a call to
* H5Eget_auto2() will fail and will indicate that the application has
* mixed H5Eset_auto1() and H5Eget_auto2(). On the other hand, mixing
* H5Eset_auto2() and H5Eget_auto1() will also cause a failure. But if
* the traversal functions are the library’s default H5Eprint1() or
* H5Eprint2(), mixing H5Eset_auto1() and H5Eget_auto2() or mixing
* H5Eset_auto2() and H5Eget_auto1() does not fail.
*
* \deprecated 1.8.0 Function H5Eget_auto() renamed to H5Eget_auto1() and
* deprecated in this release.
*/
H5_DLL herr_t H5Eget_auto1(H5E_auto1_t *func, void **client_data);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Pushes a new error record onto the error stack
*
* \param[in] file Name of the file in which the error was detected
* \param[in] func Name of the function in which the error was detected
* \param[in] line Line number in the file where the error was detected
* \param[in] maj Major error identifier
* \param[in] min Minor error identifier
* \param[in] str Error description string
* \return \herr_t
*
* \details H5Epush1() pushes a new error record onto the error stack for the
* current thread.\n
* The error has major and minor numbers \p maj_num
* and \p min_num, the function \p func where the error was detected, the
* name of the file \p file where the error was detected, the line \p line
* within that file, and an error description string \p str.\n
* The function name, filename, and error description strings must be statically
* allocated.
*
* \since 1.4.0
* \deprecated 1.8.0 Function H5Epush() renamed to H5Epush1() and
* deprecated in this release.
*/
H5_DLL herr_t H5Epush1(const char *file, const char *func, unsigned line, H5E_major_t maj, H5E_minor_t min,
const char *str);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Prints the current error stack in a default manner
*
* \param[in] stream File pointer, or \c NULL for \c stderr
* \return \herr_t
*
* \details H5Eprint1() prints prints the error stack for the current thread
* on the specified stream, \p stream. Even if the error stack is empty, a
* one-line message of the following form will be printed:
* \code{.unparsed}
* HDF5-DIAG: Error detected in thread 0.
* \endcode
* H5Eprint1() is a convenience function for H5Ewalk1() with a function
* that prints error messages. Users are encouraged to write their own
* more specific error handlers.
*
* \deprecated 1.8.0 Function H5Eprint() renamed to H5Eprint1() and
* deprecated in this release.
*/
H5_DLL herr_t H5Eprint1(FILE *stream);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Turns automatic error printing on or off
*
* \param[in] func Function to be called upon an error condition
* \param[in] client_data Data passed to the error function
* \return \herr_t
*
* \details H5Eset_auto1() turns on or off automatic printing of errors. When
* turned on (non-null \p func pointer), any API function which returns
* an error indication will first call \p func, passing it \p
* client_data as an argument.
*
* \p func, a function conforming to the #H5E_auto1_t prototype, is
* defined in the H5Epublic.h source code file as:
* \snippet this H5E_auto1_t_snip
*
* When the library is first initialized, the auto printing function is
* set to H5Eprint1() (cast appropriately) and \p client_data is the
* standard error stream pointer, \c stderr.
*
* Automatic stack traversal is always in the #H5E_WALK_DOWNWARD
* direction.
*
* \deprecated 1.8.0 Function H5Eset_auto() renamed to H5Eset_auto1() and
* deprecated in this release.
*/
H5_DLL herr_t H5Eset_auto1(H5E_auto1_t func, void *client_data);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Walks the current error stack, calling the specified function
*
* \param[in] direction Direction in which the error stack is to be walked
* \param[in] func Function to be called for each error encountered
* \param[in] client_data Data to be passed to \p func
* \return \herr_t
*
* \details H5Ewalk1() walks the error stack for the current thread and calls
* the function specified in \p func for each error along the way.
*
* \p direction specifies whether the stack is walked from the inside
* out or the outside in. A value of #H5E_WALK_UPWARD means to begin
* with the most specific error and end at the API; a value of
* #H5E_WALK_DOWNWARD means to start at the API and end at the
* innermost function where the error was first detected.
*
* \p func, a function conforming to the #H5E_walk1_t prototype, will
* be called for each error in the error stack. Its arguments will
* include an index number \c n (beginning at zero regardless of stack
* traversal direction), an error stack entry \c err_desc, and the \c
* client_data pointer passed to H5Eprint(). The #H5E_walk1_t prototype
* is as follows:
* \snippet this H5E_walk1_t_snip
*
* \deprecated 1.8.0 Function H5Ewalk() renamed to H5Ewalk1() and
* deprecated in this release.
*/
H5_DLL herr_t H5Ewalk1(H5E_direction_t direction, H5E_walk1_t func, void *client_data);
H5_DLL char * H5Eget_major(H5E_major_t maj);
H5_DLL char * H5Eget_minor(H5E_minor_t min);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Returns a character string describing an error specified by a major
* error number
*
* \param[in] maj Major error number
* \return \herr_t
*
* \details Given a major error number, H5Eget_major() returns a constant
* character string that describes the error.
*
* \attention This function returns a dynamically allocated string (\c char
* array). An application calling this function must free the memory
* associated with the return value to prevent a memory leak.
*
* \deprecated 1.8.0 Function deprecated in this release.
*/
H5_DLL char *H5Eget_major(H5E_major_t maj);
/**
* --------------------------------------------------------------------------
* \ingroup H5E
*
* \brief Returns a character string describing an error specified by a minor
* error number
*
* \param[in] min Minor error number
* \return \herr_t
*
* \details Given a minor error number, H5Eget_minor() returns a constant
* character string that describes the error.
*
* \attention In the Release 1.8.x series, H5Eget_minor() returns a string of
* dynamic allocated \c char array. An application calling this
* function from an HDF5 library of Release 1.8.0 or later must free
* the memory associated with the return value to prevent a memory
* leak. This is a change from the 1.6.x release series.
*
* \deprecated 1.8.0 Function deprecated and return type changed in this release.
*/
H5_DLL char *H5Eget_minor(H5E_minor_t min);
#endif /* H5_NO_DEPRECATED_SYMBOLS */
#ifdef __cplusplus
src/H5FDcore.h
+63 -1
View File
@@ -25,8 +25,70 @@
#ifdef __cplusplus
extern "C" {
#endif
H5_DLL hid_t H5FD_core_init(void);
H5_DLL hid_t H5FD_core_init(void);
/**
* \ingroup FAPL
*
* \brief Modifies the file access property list to use the #H5FD_CORE driver
*
* \fapl_id
* \param[in] increment Size, in bytes, of memory increments
* \param[in] backing_store Boolean flag indicating whether to write the file
* contents to disk when the file is closed
* \returns \herr_t
*
* \details H5Pset_fapl_core() modifies the file access property list to use the
* #H5FD_CORE driver.
*
* The #H5FD_CORE driver enables an application to work with a file in
* memory, speeding reads and writes as no disk access is made. File
* contents are stored only in memory until the file is closed. The \p
* backing_store parameter determines whether file contents are ever
* written to disk.
*
* \p increment specifies the increment by which allocated memory is to
* be increased each time more memory is required.
*
* While using H5Fcreate() to create a core file, if the \p
* backing_store is set to 1 (TRUE), the file contents are flushed to a
* file with the same name as this core file when the file is closed or
* access to the file is terminated in memory.
*
* The application is allowed to open an existing file with #H5FD_CORE
* driver. While using H5Fopen() to open an existing file, if the \p
* backing_store is set to 1 (TRUE) and the \c flags for H5Fopen() is set to
* #H5F_ACC_RDWR, any change to the file contents are saved to the file
* when the file is closed. If \p backing_store is set to 0 (FALSE) and the \c
* flags for H5Fopen() is set to #H5F_ACC_RDWR, any change to the file
* contents will be lost when the file is closed. If the flags for
* H5Fopen() is set to #H5F_ACC_RDONLY, no change to the file is
* allowed either in memory or on file.
*
* \note Currently this driver cannot create or open family or multi files.
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_fapl_core(hid_t fapl_id, size_t increment, hbool_t backing_store);
/**
* \ingroup FAPL
*
* \brief Queries core file driver properties
*
* \fapl_id
* \param[out] increment Size, in bytes, of memory increments
* \param[out] backing_store Boolean flag indicating whether to write the file
* contents to disk when the file is closed
* \returns \herr_t
*
* \details H5Pget_fapl_core() queries the #H5FD_CORE driver properties as set
* by H5Pset_fapl_core().
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pget_fapl_core(hid_t fapl_id, size_t *increment /*out*/, hbool_t *backing_store /*out*/);
#ifdef __cplusplus
}
src/H5FDdirect.h
+62 -1
View File
@@ -37,8 +37,69 @@ extern "C" {
#define FBSIZE_DEF 4096
#define CBSIZE_DEF 16 * 1024 * 1024
H5_DLL hid_t H5FD_direct_init(void);
H5_DLL hid_t H5FD_direct_init(void);
/**
* \ingroup FAPL
*
* \brief Sets up use of the direct I/O driver
*
* \fapl_id
* \param[in] alignment Required memory alignment boundary
* \param[in] block_size File system block size
* \param[in] cbuf_size Copy buffer size
* \returns \herr_t
*
* \details H5Pset_fapl_direct() sets the file access property list, \p fapl_id,
* to use the direct I/O driver, #H5FD_DIRECT. With this driver, data
* is written to or read from the file synchronously without being
* cached by the system.
*
* File systems usually require the data address in memory, the file
* address, and the size of the data to be aligned. The HDF5 library’s
* direct I/O driver is able to handle unaligned data, though that will
* consume some additional memory resources and may slow
* performance. To get better performance, use the system function \p
* posix_memalign to align the data buffer in memory and the HDF5
* function H5Pset_alignment() to align the data in the file. Be aware,
* however, that aligned data I/O may cause the HDF5 file to be bigger
* than the actual data size would otherwise require because the
* alignment may leave some holes in the file.
*
* \p alignment specifies the required alignment boundary in memory.
*
* \p block_size specifies the file system block size. A value of 0
* (zero) means to use HDF5 library’s default value of 4KB.
*
* \p cbuf_size specifies the copy buffer size.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pset_fapl_direct(hid_t fapl_id, size_t alignment, size_t block_size, size_t cbuf_size);
/**
* \ingroup FAPL
*
* \brief Retrieves direct I/O driver settings
*
* \fapl_id
* \param[out] boundary Required memory alignment boundary
* \param[out] block_size File system block size
* \param[out] cbuf_size Copy buffer size
* \returns \herr_t
*
* \details H5Pget_fapl_direct() retrieves the required memory alignment (\p
* alignment), file system block size (\p block_size), and copy buffer
* size (\p cbuf_size) settings for the direct I/O driver, #H5FD_DIRECT,
* from the file access property list \p fapl_id.
*
* See H5Pset_fapl_direct() for discussion of these values,
* requirements, and important considerations.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pget_fapl_direct(hid_t fapl_id, size_t *boundary /*out*/, size_t *block_size /*out*/,
size_t *cbuf_size /*out*/);
src/H5FDfamily.h
+51 -1
View File
@@ -26,8 +26,58 @@
extern "C" {
#endif
H5_DLL hid_t H5FD_family_init(void);
H5_DLL hid_t H5FD_family_init(void);
/**
* \ingroup FAPL
*
* \brief Sets the file access property list to use the family driver
*
* \fapl_id
* \param[in] memb_size Size in bytes of each file member
* \param[in] memb_fapl_id Identifier of file access property list for
* each family member
* \returns \herr_t
*
* \details H5Pset_fapl_family() sets the file access property list identifier,
* \p fapl_id, to use the family driver.
*
* \p memb_size is the size in bytes of each file member. This size
* will be saved in file when the property list \p fapl_id is used to
* create a new file. If \p fapl_id is used to open an existing file,
* \p memb_size has to be equal to the original size saved in file. A
* failure with an error message indicating the correct member size
* will be returned if \p memb_size does not match the size saved. If
* any user does not know the original size, #H5F_FAMILY_DEFAULT can be
* passed in. The library will retrieve the saved size.
*
* \p memb_fapl_id is the identifier of the file access property list
* to be used for each family member.
*
* \version 1.8.0 Behavior of the \p memb_size parameter was changed.
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_fapl_family(hid_t fapl_id, hsize_t memb_size, hid_t memb_fapl_id);
/**
* \ingroup FAPL
*
* \brief Returns file access property list information
*
* \fapl_id
* \param[out] memb_size Size in bytes of each file member
* \param[out] memb_fapl_id Identifier of file access property list for
* each family member
* \returns \herr_t
*
* \details H5Pget_fapl_family() returns file access property list for use with
* the family driver. This information is returned through the output
* parameters.
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pget_fapl_family(hid_t fapl_id, hsize_t *memb_size /*out*/, hid_t *memb_fapl_id /*out*/);
#ifdef __cplusplus
src/H5FDhdfs.h
+13 -1
View File
@@ -112,8 +112,20 @@ typedef struct H5FD_hdfs_fapl_t {
int32_t stream_buffer_size;
} H5FD_hdfs_fapl_t;
H5_DLL hid_t H5FD_hdfs_init(void);
H5_DLL hid_t H5FD_hdfs_init(void);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pget_fapl_hdfs(hid_t fapl_id, H5FD_hdfs_fapl_t *fa_out);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pset_fapl_hdfs(hid_t fapl_id, H5FD_hdfs_fapl_t *fa);
#ifdef __cplusplus
src/H5FDlog.h
+404 -1
View File
@@ -65,7 +65,410 @@
extern "C" {
#endif
H5_DLL hid_t H5FD_log_init(void);
H5_DLL hid_t H5FD_log_init(void);
/**
* \ingroup FAPL
*
* \brief Sets up the logging virtual file driver (#H5FD_LOG) for use
*
* \fapl_id
* \param[in] logfile Name of the log file
* \param[in] flags Flags specifying the types of logging activity
* \param[in] buf_size The size of the logging buffers, in bytes (see description)
* \returns \herr_t
*
* \details H5Pset_fapl_log() modifies the file access property list to use the
* logging driver, #H5FD_LOG. The logging virtual file driver (VFD) is
* a clone of the standard SEC2 (#H5FD_SEC2) driver with additional
* facilities for logging VFD metrics and activity to a file.
*
* \p logfile is the name of the file in which the logging entries are
* to be recorded.
*
* The actions to be logged are specified in the parameter \p flags
* using the pre-defined constants described in the following
* table. Multiple flags can be set through the use of a logical \c OR
* contained in parentheses. For example, logging read and write
* locations would be specified as
* \Code{(H5FD_LOG_LOC_READ|H5FD_LOG_LOC_WRITE)}.
*
* <table>
* <caption>Table1: Logging Flags</caption>
* <tr>
* <td>
* #H5FD_LOG_LOC_READ
* </td>
* <td rowspan="3">
* Track the location and length of every read, write, or seek operation.
* </td>
* </tr>
* <tr><td>#H5FD_LOG_LOC_WRITE</td></tr>
* <tr><td>#H5FD_LOG_LOC_SEEK</td></tr>
* <tr>
* <td>
* #H5FD_LOG_LOC_IO
* </td>
* <td>
* Track all I/O locations and lengths. The logical equivalent of the following:
* \Code{(#H5FD_LOG_LOC_READ | #H5FD_LOG_LOC_WRITE | #H5FD_LOG_LOC_SEEK)}
* </td>
* </tr>
* <tr>
* <td>
* #H5FD_LOG_FILE_READ
* </td>
* <td rowspan="2">
* Track the number of times each byte is read or written.
* </td>
* </tr>
* <tr><td>#H5FD_LOG_FILE_WRITE</td></tr>
* <tr>
* <td>
* #H5FD_LOG_FILE_IO
* </td>
* <td>
* Track the number of times each byte is read and written. The logical
* equivalent of the following:
* \Code{(#H5FD_LOG_FILE_READ | #H5FD_LOG_FILE_WRITE)}
* </td>
* </tr>
* <tr>
* <td>
* #H5FD_LOG_FLAVOR
* </td>
* <td>
* Track the type, or flavor, of information stored at each byte.
* </td>
* </tr>
* <tr>
* <td>
* #H5FD_LOG_NUM_READ
* </td>
* <td rowspan="4">
* Track the total number of read, write, seek, or truncate operations that occur.
* </td>
* </tr>
* <tr><td>#H5FD_LOG_NUM_WRITE</td></tr>
* <tr><td>#H5FD_LOG_NUM_SEEK</td></tr>
* <tr><td>#H5FD_LOG_NUM_TRUNCATE</td></tr>
* <tr>
* <td>
* #H5FD_LOG_NUM_IO
* </td>
* <td>
* Track the total number of all types of I/O operations. The logical equivalent
* of the following:
* \Code{(#H5FD_LOG_NUM_READ | #H5FD_LOG_NUM_WRITE | #H5FD_LOG_NUM_SEEK | #H5FD_LOG_NUM_TRUNCATE)}
* </td>
* </tr>
* <tr>
* <td>
* #H5FD_LOG_TIME_OPEN
* </td>
* <td rowspan="6">
* Track the time spent in open, stat, read, write, seek, or close operations.
* </td>
* </tr>
* <tr><td>#H5FD_LOG_TIME_STAT</td></tr>
* <tr><td>#H5FD_LOG_TIME_READ</td></tr>
* <tr><td>#H5FD_LOG_TIME_WRITE</td></tr>
* <tr><td>#H5FD_LOG_TIME_SEEK</td></tr>
* <tr><td>#H5FD_LOG_TIME_CLOSE</td></tr>
* <tr>
* <td>
* #H5FD_LOG_TIME_IO
* </td>
* <td>
* Track the time spent in each of the above operations. The logical equivalent
* of the following:
* \Code{(#H5FD_LOG_TIME_OPEN | #H5FD_LOG_TIME_STAT | #H5FD_LOG_TIME_READ | #H5FD_LOG_TIME_WRITE |
* #H5FD_LOG_TIME_SEEK | #H5FD_LOG_TIME_CLOSE)}
* </td>
* </tr>
* <tr>
* <td>
* #H5FD_LOG_ALLOC
* </td>
* <td>
* Track the allocation of space in the file.
* </td>
* </tr>
* <tr>
* <td>
* #H5FD_LOG_ALL
* </td>
* <td>
* Track everything. The logical equivalent of the following:
* \Code{(#H5FD_LOG_ALLOC | #H5FD_LOG_TIME_IO | #H5FD_LOG_NUM_IO | #H5FD_LOG_FLAVOR | #H5FD_LOG_FILE_IO |
* #H5FD_LOG_LOC_IO)}
* </td>
* </tr>
* </table>
* The logging driver can track the number of times each byte in the file is
* read from or written to (using #H5FD_LOG_FILE_READ and #H5FD_LOG_FILE_WRITE)
* and what kind of data is at that location (e.g., metadata, raw data; using
* #H5FD_LOG_FLAVOR). This information is tracked in internal buffers of size
* buf_size, which must be at least the maximum size in bytes of the file to be
* logged while the log driver is in use.\n
* One buffer of size buf_size will be created for each of #H5FD_LOG_FILE_READ,
* #H5FD_LOG_FILE_WRITE and #H5FD_LOG_FLAVOR when those flags are set; these
* buffers will not grow as the file increases in size.
*
* \par Output:
* This section describes the logging driver (LOG VFD) output.\n
* The table, immediately below, describes output of the various logging driver
* flags and function calls. A list of valid flavor values, describing the type
* of data stored, follows the table.
* <table>
* <caption>Table2: Logging Output</caption>
* <tr>
* <th>Flag</th><th>VFD Call</th><th>Output and Comments</th>
* </th>
* </tr>
* <tr>
* <td>#H5FD_LOG_LOC_READ</td>
* <td>Read</td>
* <td>
* \Code{%10a-%10a (%10Zu bytes) (%s) Read}\n\n
* Start position\n
* End position\n
* Number of bytes\n
* Flavor of read\n\n
* Adds \Code{(\%f s)} and seek time if #H5FD_LOG_TIME_SEEK is also set.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_LOC_READ</td>
* <td>Read Error</td>
* <td>
* \Code{Error! Reading: %10a-%10a (%10Zu bytes)}\n\n
* Same parameters as non-error entry.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_LOC_WRITE</td>
* <td>Write</td>
* <td>
* \Code{%10a-%10a (%10Zu bytes) (%s) Written}\n\n
* Start position\n
* End position\n
* Number of bytes\n
* Flavor of write\n\n
* Adds \Code{(\%f s)} and seek time if #H5FD_LOG_TIME_SEEK is also set.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_LOC_WRITE</td>
* <td>Write Error</td>
* <td>
* \Code{Error! Writing: %10a-%10a (%10Zu bytes)}\n\n
* Same parameters as non-error entry.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_LOC_SEEK</td>
* <td>Read, Write</td>
* <td>
* \Code{Seek: From %10a-%10a}\n\n
* Start position\n
* End position\n\n
* Adds \Code{(\%f s)} and seek time if #H5FD_LOG_TIME_SEEK is also set.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_FILE_READ</td>
* <td>Close</td>
* <td>
* Begins with:\n
* Dumping read I/O information\n\n
* Then, for each range of identical values, there is this line:\n
* \Code{Addr %10-%10 (%10lu bytes) read from %3d times}\n\n
* Start address\n
* End address\n
* Number of bytes\n
* Number of times read\n\n
* Note: The data buffer is scanned and each range of identical values
* gets one entry in the log file to save space and make it easier to read.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_FILE_WRITE</td>
* <td>Close</td>
* <td>
* Begins with:\n
* Dumping read I/O information\n\n
* Then, for each range of identical values, there is this line:\n
* \Code{Addr %10-%10 (%10lu bytes) written to %3d times}\n\n
* Start address\n
* End address\n
* Number of bytes\n
* Number of times written\n\n
* Note: The data buffer is scanned and each range of identical values
* gets one entry in the log file to save space and make it easier to read.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_FLAVOR</td>
* <td>Close</td>
* <td>
* Begins with:\n
* Dumping I/O flavor information\n\n
* Then, for each range of identical values, there is this line:\n
* \Code{Addr %10-%10 (%10lu bytes) flavor is %s}\n\n
* Start address\n
* End address\n
* Number of bytes\n
* Flavor\n\n
* Note: The data buffer is scanned and each range of identical values
* gets one entry in the log file to save space and make it easier to read.
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_NUM_READ</td>
* <td>Close</td>
* <td>
* Total number of read operations: \Code{%11u}
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_NUM_WRITE</td>
* <td>Close</td>
* <td>
* Total number of write operations: \Code{%11u}
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_NUM_SEEK</td>
* <td>Close</td>
* <td>
* Total number of seek operations: \Code{%11u}
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_NUM_TRUNCATE</td>
* <td>Close</td>
* <td>
* Total number of truncate operations: \Code{%11u}
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_TIME_OPEN</td>
* <td>Open</td>
* <td>
* Open took: \Code{(\%f s)}
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_TIME_READ</td>
* <td>Close, Read</td>
* <td>
* Total time in read operations: \Code{\%f s}\n\n
* See also: #H5FD_LOG_LOC_READ
* </td>
* </tr>
* </tr>
* <tr>
* <td>#H5FD_LOG_TIME_WRITE</td>
* <td>Close, Write</td>
* <td>
* Total time in write operations: \Code{\%f s}\n\n
* See also: #H5FD_LOG_LOC_WRITE
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_TIME_SEEK</td>
* <td>Close, Read, Write</td>
* <td>
* Total time in write operations: \Code{\%f s}\n\n
* See also: #H5FD_LOG_LOC_SEEK or #H5FD_LOG_LOC_WRITE
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_TIME_CLOSE</td>
* <td>Close</td>
* <td>
* Close took: \Code{(\%f s)}
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_TIME_STAT</td>
* <td>Open</td>
* <td>
* Stat took: \Code{(\%f s)}
* </td>
* </tr>
* <tr>
* <td>#H5FD_LOG_ALLOC</td>
* <td>Alloc</td>
* <td>
* \Code{%10-%10 (%10Hu bytes) (\%s) Allocated}\n\n
* Start of address space\n
* End of address space\n
* Total size allocation\n
* Flavor of allocation
* </td>
* </tr>
* </table>
*
* \par Flavors:
* The \Emph{flavor} describes the type of stored information. The following
* table lists the flavors that appear in log output and briefly describes each.
* These terms are provided here to aid in the construction of log message
* parsers; a full description is beyond the scope of this document.
* <table>
* <caption>Table3: Flavors of logged data</caption>
* <tr>
* <th>Flavor</th><th>Description</th>
* </th>
* </tr>
* <tr>
* <td>#H5FD_MEM_NOLIST</td>
* <td>Error value</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_DEFAULT</td>
* <td>Value not yet set.\n
* May also be a datatype set in a larger allocation that will be
* suballocated by the library.</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_SUPER</td>
* <td>Superblock data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_BTREE</td>
* <td>B-tree data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_DRAW</td>
* <td>Raw data (for example, contents of a dataset)</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_GHEAP</td>
* <td>Global heap data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_LHEAP</td>
* <td>Local heap data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_OHDR</td>
* <td>Object header data</td>
* </tr>
* </table>
*
* \version 1.8.7 The flags parameter has been changed from \Code{unsigned int}
* to \Code{unsigned long long}.
* The implementation of the #H5FD_LOG_TIME_OPEN, #H5FD_LOG_TIME_READ,
* #H5FD_LOG_TIME_WRITE, and #H5FD_LOG_TIME_SEEK flags has been finished.
* New flags were added: #H5FD_LOG_NUM_TRUNCATE and #H5FD_LOG_TIME_STAT.
* \version 1.6.0 The \c verbosity parameter has been removed.
* Two new parameters have been added: \p flags of type \Code{unsigned} and
* \p buf_size of type \Code{size_t}.
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_fapl_log(hid_t fapl_id, const char *logfile, unsigned long long flags, size_t buf_size);
#ifdef __cplusplus
src/H5FDmirror.h
+13 -1
View File
@@ -61,8 +61,20 @@ typedef struct H5FD_mirror_fapl_t {
char remote_ip[H5FD_MIRROR_MAX_IP_LEN + 1];
} H5FD_mirror_fapl_t;
H5_DLL hid_t H5FD_mirror_init(void);
H5_DLL hid_t H5FD_mirror_init(void);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pget_fapl_mirror(hid_t fapl_id, H5FD_mirror_fapl_t *fa_out);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pset_fapl_mirror(hid_t fapl_id, H5FD_mirror_fapl_t *fa);
#ifdef __cplusplus
src/H5FDmpi.h
+5 -3
View File
@@ -34,10 +34,12 @@
*/
#define H5D_MULTI_CHUNK_IO_COL_THRESHOLD 60
/* Type of I/O for data transfer properties */
/**
* Type of I/O for data transfer properties
*/
typedef enum H5FD_mpio_xfer_t {
H5FD_MPIO_INDEPENDENT = 0, /*zero is the default*/
H5FD_MPIO_COLLECTIVE
H5FD_MPIO_INDEPENDENT = 0, /**< Use independent I/O access */
H5FD_MPIO_COLLECTIVE /**< Use collective I/O access */
} H5FD_mpio_xfer_t;
/* Type of chunked dataset I/O */
src/H5FDmpio.h
+224 -1
View File
@@ -44,14 +44,237 @@ H5_DLLVAR hbool_t H5FD_mpi_opt_types_g;
#ifdef __cplusplus
extern "C" {
#endif
H5_DLL hid_t H5FD_mpio_init(void);
H5_DLL hid_t H5FD_mpio_init(void);
/**
* \ingroup FAPL
*
* \brief Stores MPI IO communicator information to the file access property list
*
* \fapl_id
* \param[in] comm MPI-2 communicator
* \param[in] info MPI-2 info object
* \returns \herr_t
*
* \details H5Pset_fapl_mpio() stores the user-supplied MPI IO parameters \p
* comm, for communicator, and \p info, for information, in the file
* access property list \p fapl_id. That property list can then be used
* to create and/or open a file.
*
* H5Pset_fapl_mpio() is available only in the parallel HDF5 library
* and is not a collective function.
*
* \p comm is the MPI communicator to be used for file open, as defined
* in \c MPI_File_open of MPI-2. This function makes a duplicate of the
* communicator, so modifications to \p comm after this function call
* returns have no effect on the file access property list.
*
* \p info is the MPI Info object to be used for file open, as defined
* in MPI_File_open() of MPI-2. This function makes a duplicate copy of
* the Info object, so modifications to the Info object after this
* function call returns will have no effect on the file access
* property list.
*
* If the file access property list already contains previously-set
* communicator and Info values, those values will be replaced and the
* old communicator and Info object will be freed.
*
* \note Raw dataset chunk caching is not currently supported when using this
* file driver in read/write mode. All calls to H5Dread() and H5Dwrite()
* will access the disk directly, and H5Pset_cache() and
* H5Pset_chunk_cache() will have no effect on performance.\n
* Raw dataset chunk caching is supported when this driver is used in
* read-only mode.
*
* \version 1.4.5 Handling of the MPI Communicator and Info object changed at
* this release. A duplicate of each of these is now stored in the property
* list instead of pointers to each.
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_fapl_mpio(hid_t fapl_id, MPI_Comm comm, MPI_Info info);
/**
* \ingroup FAPL
*
* \brief Returns MPI IO communicator information
*
* \fapl_id
* \param[out] comm MPI-2 communicator
* \param[out] info MPI-2 info object
* \returns \herr_t
*
* \details If the file access property list is set to the #H5FD_MPIO driver,
* H5Pget_fapl_mpio() returns duplicates of the stored MPI communicator
* and Info object through the \p comm and \p info pointers, if those
* values are non-null.
*
* Since the MPI communicator and Info object are duplicates of the
* stored information, future modifications to the access property list
* will not affect them. It is the responsibility of the application to
* free these objects.
*
* \version 1.4.5 Handling of the MPI Communicator and Info object changed at
* this release. A duplicate of each of these is now stored in the
* property list instead of pointers to each.
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pget_fapl_mpio(hid_t fapl_id, MPI_Comm *comm /*out*/, MPI_Info *info /*out*/);
/**
* \ingroup DXPL
*
* \brief Sets data transfer mode
*
* \dxpl_id
* \param[in] xfer_mode Transfer mode
* \returns \herr_t
*
* \details H5Pset_dxpl_mpio() sets the data transfer property list \p dxpl_id
* to use transfer mode \p xfer_mode. The property list can then be
* used to control the I/O transfer mode during data I/O operations.
*
* Valid transfer modes are #H5FD_MPIO_INDEPENDENT (default) and
* #H5FD_MPIO_COLLECTIVE.
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_dxpl_mpio(hid_t dxpl_id, H5FD_mpio_xfer_t xfer_mode);
/**
* \ingroup DXPL
*
* \brief Returns the data transfer mode
*
* \dxpl_id
* \param[out] xfer_mode Transfer mode
* \returns \herr_t
*
* \details H5Pget_dxpl_mpio() queries the data transfer mode currently set in
* the data transfer property list \p dxpl_id.
*
* Upon return, \p xfer_mode contains the data transfer mode, if it is
* non-null.
*
* H5Pget_dxpl_mpio() is not a collective function.
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pget_dxpl_mpio(hid_t dxpl_id, H5FD_mpio_xfer_t *xfer_mode /*out*/);
/**
* \ingroup DXPL
*
* \brief Sets data transfer mode
*
* \dxpl_id
* \param[in] opt_mode Transfer mode
* \returns \herr_t
*
* \details H5Pset_dxpl_mpio() sets the data transfer property list \p dxpl_id
* to use transfer mode xfer_mode. The property list can then be used
* to control the I/O transfer mode during data I/O operations.
*
* Valid transfer modes are #H5FD_MPIO_INDEPENDENT (default) and
* #H5FD_MPIO_COLLECTIVE.
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_dxpl_mpio_collective_opt(hid_t dxpl_id, H5FD_mpio_collective_opt_t opt_mode);
/**
* \ingroup DXPL
*
* \brief Sets a flag specifying linked-chunk I/O or multi-chunk I/O
*
* \dxpl_id
* \param[in] opt_mode Transfer mode
* \returns \herr_t
*
* \details H5Pset_dxpl_mpio_chunk_opt() specifies whether I/O is to be
* performed as linked-chunk I/O or as multi-chunk I/O. This function
* overrides the HDF5 library's internal algorithm for determining
* which mechanism to use.
*
* When an application uses collective I/O with chunked storage, the
* HDF5 library normally uses an internal algorithm to determine
* whether that I/O activity should be conducted as one linked-chunk
* I/O or as multi-chunk I/O. H5Pset_dxpl_mpio_chunk_opt() is provided
* so that an application can override the library's algorithm in
* circumstances where the library might lack the information needed to
* make an optimal decision.
*
* H5Pset_dxpl_mpio_chunk_opt() works by setting one of the following
* flags in the parameter \p opt_mode:
* - #H5FD_MPIO_CHUNK_ONE_IO - Do one-link chunked I/O
* - #H5FD_MPIO_CHUNK_MULTI_IO - Do multi-chunked I/O
*
* This function works by setting a corresponding property in the
* dataset transfer property list \p dxpl_id.
*
* The library performs I/O in the specified manner unless it
* determines that the low-level MPI IO package does not support the
* requested behavior; in such cases, the HDF5 library will internally
* use independent I/O.
*
* Use of this function is optional.
*
* \todo Add missing version information
*
*/
H5_DLL herr_t H5Pset_dxpl_mpio_chunk_opt(hid_t dxpl_id, H5FD_mpio_chunk_opt_t opt_mode);
/**
* \ingroup DXPL
*
* \brief Sets a numeric threshold for linked-chunk I/O
*
* \dxpl_id
* \param[in] num_chunk_per_proc
* \returns \herr_t
*
* \details H5Pset_dxpl_mpio_chunk_opt_num() sets a numeric threshold for the
* use of linked-chunk I/O.
*
* The library will calculate the average number of chunks selected by
* each process when doing collective access with chunked storage. If
* the number is greater than the threshold set in \p
* num_chunk_per_proc, the library will use linked-chunk I/O;
* otherwise, a separate I/O process will be invoked for each chunk
* (multi-chunk I/O).
*
* \todo Add missing version information
*
*/
H5_DLL herr_t H5Pset_dxpl_mpio_chunk_opt_num(hid_t dxpl_id, unsigned num_chunk_per_proc);
/**
* \ingroup DXPL
*
* \brief Sets a ratio threshold for collective I/O
*
* \dxpl_id
* \param[in] percent_num_proc_per_chunk
* \returns \herr_t
*
* \details H5Pset_dxpl_mpio_chunk_opt_ratio() sets a threshold for the use of
* collective I/O based on the ratio of processes with collective
* access to a dataset with chunked storage. The decision whether to
* use collective I/O is made on a per-chunk basis.
*
* The library will calculate the percentage of the total number of
* processes, the ratio, that hold selections in each chunk. If that
* percentage is greater than the threshold set in \p
* percent_proc_per_chunk, the library will do collective I/O for this
* chunk; otherwise, independent I/O will be done for the chunk.
*
* \todo Add missing version information
*
*/
H5_DLL herr_t H5Pset_dxpl_mpio_chunk_opt_ratio(hid_t dxpl_id, unsigned percent_num_proc_per_chunk);
#ifdef __cplusplus
}
src/H5FDmulti.h
+218 -1
View File
@@ -25,11 +25,228 @@
#ifdef __cplusplus
extern "C" {
#endif
H5_DLL hid_t H5FD_multi_init(void);
H5_DLL hid_t H5FD_multi_init(void);
/**
* \ingroup FAPL
*
* \brief Sets up use of the multi-file driver
*
* \fapl_id
* \param[in] memb_map Maps memory usage types to other memory usage types
* \param[in] memb_fapl Property list for each memory usage type
* \param[in] memb_name Name generator for names of member files
* \param[in] memb_addr The offsets within the virtual address space, from 0
* (zero) to #HADDR_MAX, at which each type of data storage begins
* \param[in] relax Allows read-only access to incomplete file sets when \c TRUE
* \returns \herr_t
*
* \details H5Pset_fapl_multi() sets the file access property list \p fapl_id to
* use the multi-file driver.
*
* The multi-file driver enables different types of HDF5 data and
* metadata to be written to separate files. These files are viewed by
* the HDF5 library and the application as a single virtual HDF5 file
* with a single HDF5 file address space. The types of data that can be
* broken out into separate files include raw data, the superblock,
* B-tree data, global heap data, local heap data, and object
* headers. At the programmer's discretion, two or more types of data
* can be written to the same file while other types of data are
* written to separate files.
*
* The array \p memb_map maps memory usage types to other memory usage
* types and is the mechanism that allows the caller to specify how
* many files are created. The array contains #H5FD_MEM_NTYPES entries,
* which are either the value #H5FD_MEM_DEFAULT or a memory usage
* type. The number of unique values determines the number of files
* that are opened.
*
* The array \p memb_fapl contains a property list for each memory
* usage type that will be associated with a file.
*
* The array \p memb_name should be a name generator (a
* \Code{printf}-style format with a \Code{%s} which will be replaced
* with the name passed to H5FDopen(), usually from H5Fcreate() or
* H5Fopen()).
*
* The array \p memb_addr specifies the offsets within the virtual
* address space, from 0 (zero) to #HADDR_MAX, at which each type of
* data storage begins.
*
* If \p relax is set to 1 (TRUE), then opening an existing file for
* read-only access will not fail if some file members are
* missing. This allows a file to be accessed in a limited sense if
* just the meta data is available.
*
* Default values for each of the optional arguments are as follows:
* <table>
* <tr>
* <td>\p memb_map</td>
* <td>The default member map contains the value #H5FD_MEM_DEFAULT for each element.</td>
* </tr>
* <tr>
* <td>
* \p memb_fapl
* </td>
* <td>
* The default value is #H5P_DEFAULT for each element.
* </td>
* </tr>
* <tr>
* <td>
* \p memb_name
* </td>
* <td>
* The default string is \Code{%s-X.h5} where \c X is one of the following letters:
* - \c s for #H5FD_MEM_SUPER
* - \c b for #H5FD_MEM_BTREE
* - \c r for #H5FD_MEM_DRAW
* - \c g for #H5FD_MEM_GHEAP
* - \c l for #H5FD_MEM_LHEAP
* - \c o for #H5FD_MEM_OHDR
* </td>
* </tr>
* <tr>
* <td>
* \p memb_addr
* </td>
* <td>
* The default setting is that the address space is equally divided
* among all of the elements:
* - #H5FD_MEM_SUPER \Code{-> 0 * (HADDR_MAX/6)}
* - #H5FD_MEM_BTREE \Code{-> 1 * (HADDR_MAX/6)}
* - #H5FD_MEM_DRAW \Code{-> 2 * (HADDR_MAX/6)}
* - #H5FD_MEM_GHEAP \Code{-> 3 * (HADDR_MAX/6)}
* - #H5FD_MEM_LHEAP \Code{-> 4 * (HADDR_MAX/6)}
* - #H5FD_MEM_OHDR \Code{-> 5 * (HADDR_MAX/6)}
* </td>
* </tr>
* </table>
*
* \par Example:
* The following code sample sets up a multi-file access property list that
* partitions data into meta and raw files, each being one-half of the address:\n
* \code
* H5FD_mem_t mt, memb_map[H5FD_MEM_NTYPES];
* hid_t memb_fapl[H5FD_MEM_NTYPES];
* const char *memb[H5FD_MEM_NTYPES];
* haddr_t memb_addr[H5FD_MEM_NTYPES];
*
* // The mapping...
* for (mt=0; mt<H5FD_MEM_NTYPES; mt++) {
* memb_map[mt] = H5FD_MEM_SUPER;
* }
* memb_map[H5FD_MEM_DRAW] = H5FD_MEM_DRAW;
*
* // Member information
* memb_fapl[H5FD_MEM_SUPER] = H5P_DEFAULT;
* memb_name[H5FD_MEM_SUPER] = "%s.meta";
* memb_addr[H5FD_MEM_SUPER] = 0;
*
* memb_fapl[H5FD_MEM_DRAW] = H5P_DEFAULT;
* memb_name[H5FD_MEM_DRAW] = "%s.raw";
* memb_addr[H5FD_MEM_DRAW] = HADDR_MAX/2;
*
* hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
* H5Pset_fapl_multi(fapl, memb_map, memb_fapl,
* memb_name, memb_addr, TRUE);
* \endcode
*
* \version 1.6.3 \p memb_name parameter type changed to \Code{const char* const*}.
* \since 1.4.0
*/
H5_DLL herr_t H5Pset_fapl_multi(hid_t fapl_id, const H5FD_mem_t *memb_map, const hid_t *memb_fapl,
const char *const *memb_name, const haddr_t *memb_addr, hbool_t relax);
/**
* \ingroup FAPL
*
* \brief Returns information about the multi-file access property list
*
* \fapl_id
* \param[out] memb_map Maps memory usage types to other memory usage types
* \param[out] memb_fapl Property list for each memory usage type
* \param[out] memb_name Name generator for names of member files
* \param[out] memb_addr The offsets within the virtual address space, from 0
* (zero) to #HADDR_MAX, at which each type of data storage begins
* \param[out] relax Allows read-only access to incomplete file sets when \c TRUE
* \returns \herr_t
*
* \details H5Pget_fapl_multi() returns information about the multi-file access
* property list.
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pget_fapl_multi(hid_t fapl_id, H5FD_mem_t *memb_map /*out*/, hid_t *memb_fapl /*out*/,
char **memb_name /*out*/, haddr_t *memb_addr /*out*/, hbool_t *relax /*out*/);
/**
* \ingroup FAPL
*
* \brief Emulates the old split file driver
*
* \fapl_id{fapl}
* \param[in] meta_ext Metadata filename extension
* \param[in] meta_plist_id File access property list identifier for the metadata file
* \param[in] raw_ext Raw data filename extension
* \param[in] raw_plist_id
* \returns \herr_t
*
* \details H5Pset_fapl_split() is a compatibility function that enables the
* multi-file driver to emulate the split driver from HDF5 Releases 1.0
* and 1.2. The split file driver stored metadata and raw data in
* separate files but provided no mechanism for separating types of
* metadata.
*
* \p fapl is a file access property list identifier.
*
* \p meta_ext is the filename extension for the metadata file. The
* extension is appended to the name passed to H5FDopen(), usually from
* H5Fcreate() or H5Fopen(), to form the name of the metadata file. If
* the string \Code{%s} is used in the extension, it works like the
* name generator as in H5Pset_fapl_multi().
*
* \p meta_plist_id is the file access property list identifier for the
* metadata file.
*
* \p raw_ext is the filename extension for the raw data file. The
* extension is appended to the name passed to H5FDopen(), usually from
* H5Fcreate() or H5Fopen(), to form the name of the raw data file. If
* the string \Code{%s} is used in the extension, it works like the
* name generator as in H5Pset_fapl_multi().
*
* \p raw_plist_id is the file access property list identifier for the
* raw data file.
*
* If a user wishes to check to see whether this driver is in use, the
* user must call H5Pget_driver() and compare the returned value to the
* string #H5FD_MULTI. A positive match will confirm that the multi
* driver is in use; HDF5 provides no mechanism to determine whether it
* was called as the special case invoked by H5Pset_fapl_split().
*
* \par Example:
* \code
* // Example 1: Both metadata and rawdata files are in the same
* // directory. Use Station1-m.h5 and Station1-r.h5 as
* // the metadata and rawdata files.
* hid_t fapl, fid;
* fapl = H5Pcreate(H5P_FILE_ACCESS);
* H5Pset_fapl_split(fapl, "-m.h5", H5P_DEFAULT, "-r.h5", H5P_DEFAULT);
* fid=H5Fcreate("Station1",H5F_ACC_TRUNC,H5P_DEFAULT,fapl);
*
* // Example 2: metadata and rawdata files are in different
* // directories. Use PointA-m.h5 and /pfs/PointA-r.h5 as
* // the metadata and rawdata files.
* hid_t fapl, fid;
* fapl = H5Pcreate(H5P_FILE_ACCESS);
* H5Pset_fapl_split(fapl, "-m.h5", H5P_DEFAULT, "/pfs/%s-r.h5", H5P_DEFAULT);
* fid=H5Fcreate("PointA",H5F_ACC_TRUNC,H5P_DEFAULT,fapl);
* \endcode
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_fapl_split(hid_t fapl, const char *meta_ext, hid_t meta_plist_id, const char *raw_ext,
hid_t raw_plist_id);
#ifdef __cplusplus
src/H5FDpublic.h
+86 -2
View File
@@ -329,29 +329,113 @@ struct H5FD_t {
hbool_t paged_aggr; /* Paged aggregation for file space is enabled or not */
};
/* Define enum for the source of file image callbacks */
/**
* Define enum for the source of file image callbacks
*/
//! <!-- [H5FD_file_image_op_t_snip] -->
typedef enum {
H5FD_FILE_IMAGE_OP_NO_OP,
H5FD_FILE_IMAGE_OP_PROPERTY_LIST_SET,
/**< Passed to the \p image_malloc and \p image_memcpy callbacks when a
* file image buffer is to be copied while being set in a file access
* property list (FAPL)*/
H5FD_FILE_IMAGE_OP_PROPERTY_LIST_COPY,
/**< Passed to the \p image_malloc and \p image_memcpy callbacks
* when a file image buffer is to be copied when a FAPL is copied*/
H5FD_FILE_IMAGE_OP_PROPERTY_LIST_GET,
/**<Passed to the \p image_malloc and \p image_memcpy callbacks when
* a file image buffer is to be copied while being retrieved from a FAPL*/
H5FD_FILE_IMAGE_OP_PROPERTY_LIST_CLOSE,
/**<Passed to the \p image_free callback when a file image
* buffer is to be released during a FAPL close operation*/
H5FD_FILE_IMAGE_OP_FILE_OPEN,
/**<Passed to the \p image_malloc and
* \p image_memcpy callbackswhen a
* file image buffer is to be copied during a file open operation \n
* While the file image being opened will typically be copied from a
* FAPL, this need not always be the case. For example, the core file
* driver, also known as the memory file driver, takes its initial
* image from a file.*/
H5FD_FILE_IMAGE_OP_FILE_RESIZE,
/**<Passed to the \p image_realloc callback when a file driver needs
* to resize an image buffer*/
H5FD_FILE_IMAGE_OP_FILE_CLOSE
/**<Passed to the \p image_free callback when an image buffer is to
* be released during a file close operation*/
} H5FD_file_image_op_t;
//! <!-- [H5FD_file_image_op_t_snip] -->
/* Define structure to hold file image callbacks */
/**
* Define structure to hold file image callbacks
*/
//! <!-- [H5FD_file_image_callbacks_t_snip] -->
typedef struct {
/**
* \param[in] size Size in bytes of the file image buffer to allocate
* \param[in] file_image_op A value from H5FD_file_image_op_t indicating
* the operation being performed on the file image
* when this callback is invoked
* \param[in] udata Value passed in in the H5Pset_file_image_callbacks
* parameter \p udata
*/
//! <!-- [image_malloc_snip] -->
void *(*image_malloc)(size_t size, H5FD_file_image_op_t file_image_op, void *udata);
//! <!-- [image_malloc_snip] -->
/**
* \param[in] dest Address of the destination buffer
* \param[in] src Address of the source buffer
* \param[in] file_image_op A value from #H5FD_file_image_op_t indicating
* the operation being performed on the file image
* when this callback is invoked
* \param[in] udata Value passed in in the H5Pset_file_image_callbacks
* parameter \p udata
*/
//! <!-- [image_memcpy_snip] -->
void *(*image_memcpy)(void *dest, const void *src, size_t size, H5FD_file_image_op_t file_image_op,
void *udata);
//! <!-- [image_memcpy_snip] -->
/**
* \param[in] ptr Pointer to the buffer being reallocated
* \param[in] file_image_op A value from #H5FD_file_image_op_t indicating
* the operation being performed on the file image
* when this callback is invoked
* \param[in] udata Value passed in in the H5Pset_file_image_callbacks
* parameter \p udata
*/
//! <!-- [image_realloc_snip] -->
void *(*image_realloc)(void *ptr, size_t size, H5FD_file_image_op_t file_image_op, void *udata);
//! <!-- [image_realloc_snip] -->
/**
* \param[in] udata Value passed in in the H5Pset_file_image_callbacks
* parameter \p udata
*/
//! <!-- [image_free_snip] -->
herr_t (*image_free)(void *ptr, H5FD_file_image_op_t file_image_op, void *udata);
//! <!-- [image_free_snip] -->
/**
* \param[in] udata Value passed in in the H5Pset_file_image_callbacks
* parameter \p udata
*/
//! <!-- [udata_copy_snip] -->
void *(*udata_copy)(void *udata);
//! <!-- [udata_copy_snip] -->
/**
* \param[in] udata Value passed in in the H5Pset_file_image_callbacks
* parameter \p udata
*/
//! <!-- [udata_free_snip] -->
herr_t (*udata_free)(void *udata);
//! <!-- [udata_free_snip] -->
/**
* \brief The final field in the #H5FD_file_image_callbacks_t struct,
* provides a pointer to user-defined data. This pointer will be
* passed to the image_malloc, image_memcpy, image_realloc, and
* image_free callbacks. Define udata as NULL if no user-defined
* data is provided.
*/
void *udata;
} H5FD_file_image_callbacks_t;
//! <!-- [H5FD_file_image_callbacks_t_snip] -->
#ifdef __cplusplus
extern "C" {
src/H5FDros3.h
+13 -1
View File
@@ -89,8 +89,20 @@ typedef struct H5FD_ros3_fapl_t {
extern "C" {
#endif
H5_DLL hid_t H5FD_ros3_init(void);
H5_DLL hid_t H5FD_ros3_init(void);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pget_fapl_ros3(hid_t fapl_id, H5FD_ros3_fapl_t *fa_out);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pset_fapl_ros3(hid_t fapl_id, H5FD_ros3_fapl_t *fa);
#ifdef __cplusplus
src/H5FDsplitter.h
+13 -1
View File
@@ -87,8 +87,20 @@ typedef struct H5FD_splitter_vfd_config_t {
#ifdef __cplusplus
extern "C" {
#endif
H5_DLL hid_t H5FD_splitter_init(void);
H5_DLL hid_t H5FD_splitter_init(void);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pset_fapl_splitter(hid_t fapl_id, H5FD_splitter_vfd_config_t *config_ptr);
/**
* \ingroup FAPL
*
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pget_fapl_splitter(hid_t fapl_id, H5FD_splitter_vfd_config_t *config_ptr);
#ifdef __cplusplus
src/H5FDstdio.h
+15 -1
View File
@@ -28,7 +28,21 @@
extern "C" {
#endif
H5_DLL hid_t H5FD_stdio_init(void);
H5_DLL hid_t H5FD_stdio_init(void);
/**
* \ingroup FAPL
*
* \brief Sets the standard I/O driver
*
* \fapl_id
* \returns \herr_t
*
* \details H5Pset_fapl_stdio() modifies the file access property list to use
* the standard I/O driver, H5FDstdio().
*
* \since 1.4.0
*
*/
H5_DLL herr_t H5Pset_fapl_stdio(hid_t fapl_id);
#ifdef __cplusplus
src/H5FDwindows.h
+30
View File
@@ -27,6 +27,36 @@
extern "C" {
#endif /* __cplusplus */
/**
* \ingroup FAPL
*
* \brief Sets the Windows I/O driver
*
* \fapl_id
* \returns \herr_t
*
* \details H5Pset_fapl_windows() sets the default HDF5 Windows I/O driver on
* Windows systems.
*
* Since the HDF5 library uses this driver, #H5FD_WINDOWS, by default
* on Windows systems, it is not normally necessary for a user
* application to call H5Pset_fapl_windows(). While it is not
* recommended, there may be times when a user chooses to set a
* different HDF5 driver, such as the standard I/O driver (#H5FD_STDIO)
* or the sec2 driver (#H5FD_SEC2), in a Windows
* application. H5Pset_fapl_windows() is provided so that the
* application can return to the Windows I/O driver when the time
* comes.
*
* Only the Windows driver is tested on Windows systems; other drivers
* are used at the application’s and the user’s risk.
*
* Furthermore, the Windows driver is tested and available only on
* Windows systems; it is not available on non-Windows systems.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pset_fapl_windows(hid_t fapl_id);
#ifdef __cplusplus
src/H5Fmodule.h
+28 -2
View File
@@ -31,8 +31,34 @@
/**
* \defgroup H5F H5F
* \brief File Interface
* \todo Describe concisely what the functions in this module are about.
*
* Use the functions in this module to manage HDF5 files.
*
* In the code snippets below, we show the skeletal life cycle of an HDF5 file,
* when creating a new file (left) or when opening an existing file (right).
* File creation is essentially controlled through \ref FCPL, and file access to
* new and existing files is controlled through \ref FAPL. The file \c name and
* creation or access \c mode control the interaction with the underlying
* storage such as file systems.
*
* \Emph{Proper error handling is part of the life cycle.}
* <table>
* <tr><th>Create</th><th>Open</th></tr>
* <tr valign="top">
* <td>
* \snippet H5F_examples.c life_cycle
* </td>
* <td>
* \snippet H5F_examples.c life_cycle_w_open
* </td>
* </tr>
* </table>
*
* In addition to general file management functions, there are three categories
* of functions that deal with advanced file management tasks and use cases:
* 1. The control of the HDF5 \ref MDC
* 2. The use of (MPI-) \ref PH5F HDF5
* 3. The \ref SWMR pattern
*
* \defgroup MDC Metadata Cache
* \ingroup H5F
src/H5Fpublic.h
+97 -491
View File
@@ -47,27 +47,23 @@
* We're assuming that these constants are used rather early in the hdf5
* session.
*/
#define H5F_ACC_RDONLY (H5CHECK H5OPEN 0x0000u) /**< absence of rdwr => rd-only */
#define H5F_ACC_RDWR (H5CHECK H5OPEN 0x0001u) /**< open for read and write */
#define H5F_ACC_TRUNC (H5CHECK H5OPEN 0x0002u) /**< overwrite existing files */
#define H5F_ACC_EXCL (H5CHECK H5OPEN 0x0004u) /**< fail if file already exists*/
#define H5F_ACC_RDONLY (H5CHECK H5OPEN 0x0000u) /**< Absence of RDWR: read-only */
#define H5F_ACC_RDWR (H5CHECK H5OPEN 0x0001u) /**< Open for read and write */
#define H5F_ACC_TRUNC (H5CHECK H5OPEN 0x0002u) /**< Overwrite existing files */
#define H5F_ACC_EXCL (H5CHECK H5OPEN 0x0004u) /**< Fail if file already exists*/
/* NOTE: 0x0008u was H5F_ACC_DEBUG, now deprecated */
#define H5F_ACC_CREAT (H5CHECK H5OPEN 0x0010u) /**< create non-existing files */
#define H5F_ACC_CREAT (H5CHECK H5OPEN 0x0010u) /**< Create non-existing files */
#define H5F_ACC_SWMR_WRITE \
(H5CHECK 0x0020u) /**< indicate that this file is open for writing in a \
single-writer/multi-reader (SWMR) scenario. \
Note that the process(es) opening the file for reading must \
open the file with RDONLY access, and use the special "SWMR_READ" \
access flag. */
(H5CHECK 0x0020u) /**< Indicate that this file is open for writing in a \
* single-writer/multi-reader (SWMR) scenario. \
* Note that the process(es) opening the file for reading \
* must open the file with #H5F_ACC_RDONLY and use the \
* #H5F_ACC_SWMR_READ access flag. */
#define H5F_ACC_SWMR_READ \
(H5CHECK 0x0040u) /**< indicate that this file is \
* open for reading in a \
* single-writer/multi-reader (SWMR) \
* scenario. Note that the \
* process(es) opening the file \
* for SWMR reading must also \
* open the file with the RDONLY \
* flag. */
(H5CHECK 0x0040u) /**< Indicate that this file is open for reading in a \
* single-writer/multi-reader (SWMR) scenario. Note that \
* the process(es) opening the file for SWMR reading must \
* also open the file with the #H5F_ACC_RDONLY flag. */
/**
* Default property list identifier
@@ -91,7 +87,7 @@
#define H5F_FAMILY_DEFAULT (hsize_t)0
#ifdef H5_HAVE_PARALLEL
/*
/**
* Use this constant string as the MPI_Info key to set H5Fmpio debug flags.
* To turn on H5Fmpio debug flags, set the MPI_Info value with this key to
* have the value of a string consisting of the characters that turn on the
@@ -101,11 +97,12 @@
#endif /* H5_HAVE_PARALLEL */
/**
* The difference between a single file and a set of mounted files
* The scope of an operation such as H5Fflush(), e.g.,
* a single file vs. a set of mounted files
*/
typedef enum H5F_scope_t {
H5F_SCOPE_LOCAL = 0, /**< specified file handle only */
H5F_SCOPE_GLOBAL = 1 /**< entire virtual file */
H5F_SCOPE_LOCAL = 0, /**< The specified file handle only */
H5F_SCOPE_GLOBAL = 1 /**< The entire virtual file */
} H5F_scope_t;
/**
@@ -117,16 +114,16 @@ typedef enum H5F_scope_t {
* How does file close behave?
*/
typedef enum H5F_close_degree_t {
H5F_CLOSE_DEFAULT = 0, /**< Use the degree pre-defined by underlining VFL */
H5F_CLOSE_DEFAULT = 0, /**< Use the degree pre-defined by underlying VFD */
H5F_CLOSE_WEAK = 1, /**< File closes only after all opened objects are closed */
H5F_CLOSE_SEMI = 2, /**< If no opened objects, file is close; otherwise, file close fails */
H5F_CLOSE_SEMI = 2, /**< If no opened objects, file is closed; otherwise, file close fails */
H5F_CLOSE_STRONG = 3 /**< If there are opened objects, close them first, then close file */
} H5F_close_degree_t;
/**
* Current "global" information about file
*/
//! [H5F_info2_t_snip]
//! <!-- [H5F_info2_t_snip] -->
typedef struct H5F_info2_t {
struct {
unsigned version; /**< Superblock version # */
@@ -144,7 +141,7 @@ typedef struct H5F_info2_t {
H5_ih_info_t msgs_info; /**< Shared object header message index & heap size */
} sohm;
} H5F_info2_t;
//! [H5F_info2_t_snip]
//! <!-- [H5F_info2_t_snip] -->
/**
* Types of allocation requests. The values larger than #H5FD_MEM_DEFAULT
@@ -176,12 +173,12 @@ typedef enum H5F_mem_t {
/**
* Free space section information
*/
//! [H5F_sect_info_t_snip]
//! <!-- [H5F_sect_info_t_snip] -->
typedef struct H5F_sect_info_t {
haddr_t addr; /**< Address of free space section */
hsize_t size; /**< Size of free space section */
} H5F_sect_info_t;
//! [H5F_sect_info_t_snip]
//! <!-- [H5F_sect_info_t_snip] -->
/**
* Library's format versions
@@ -193,7 +190,7 @@ typedef enum H5F_libver_t {
H5F_LIBVER_V110 = 2, /**< Use the latest v110 format for storing objects */
H5F_LIBVER_V112 = 3, /**< Use the latest v112 format for storing objects */
H5F_LIBVER_V114 = 4, /**< Use the latest v114 format for storing objects */
H5F_LIBVER_NBOUNDS
H5F_LIBVER_NBOUNDS /**< Sentinel */
} H5F_libver_t;
#define H5F_LIBVER_LATEST H5F_LIBVER_V114
@@ -201,7 +198,7 @@ typedef enum H5F_libver_t {
/**
* File space handling strategy
*/
//! [H5F_fspace_strategy_t_snip]
//! <!-- [H5F_fspace_strategy_t_snip] -->
typedef enum H5F_fspace_strategy_t {
H5F_FSPACE_STRATEGY_FSM_AGGR = 0, /**< Mechanisms: free-space managers, aggregators, and virtual file
drivers This is the library default when not set */
@@ -211,7 +208,7 @@ typedef enum H5F_fspace_strategy_t {
H5F_FSPACE_STRATEGY_NONE = 3, /**< Mechanisms: virtual file drivers */
H5F_FSPACE_STRATEGY_NTYPES /**< Sentinel */
} H5F_fspace_strategy_t;
//! [H5F_fspace_strategy_t_snip]
//! <!-- [H5F_fspace_strategy_t_snip] -->
/**
* File space handling strategy for release 1.10.0
@@ -228,7 +225,7 @@ typedef enum H5F_file_space_type_t {
H5F_FILE_SPACE_NTYPES /**< Sentinel */
} H5F_file_space_type_t;
//! [H5F_retry_info_t_snip]
//! <!-- [H5F_retry_info_t_snip] -->
#define H5F_NUM_METADATA_READ_RETRY_TYPES 21
/**
@@ -239,7 +236,7 @@ typedef struct H5F_retry_info_t {
unsigned nbins;
uint32_t *retries[H5F_NUM_METADATA_READ_RETRY_TYPES];
} H5F_retry_info_t;
//! [H5F_retry_info_t_snip]
//! <!-- [H5F_retry_info_t_snip] -->
/**
* Callback for H5Pset_object_flush_cb() in a file access property list
@@ -276,11 +273,6 @@ extern "C" {
*
*/
H5_DLL htri_t H5Fis_accessible(const char *container_name, hid_t fapl_id);
/**
* \example H5Fcreate.c
* After creating an HDF5 file with H5Fcreate(), we close it with
* H5Fclose().
*/
/**
* \ingroup H5F
*
@@ -321,7 +313,8 @@ H5_DLL htri_t H5Fis_accessible(const char *container_name, hid_t fapl_id);
* this file identifier should be closed by calling H5Fclose() when
* it is no longer needed.
*
* \include H5Fcreate.c
* \par Example
* \snippet H5F_examples.c minimal
*
* \note #H5F_ACC_TRUNC and #H5F_ACC_EXCL are mutually exclusive; use
* exactly one.
@@ -359,6 +352,11 @@ H5_DLL htri_t H5Fis_accessible(const char *container_name, hid_t fapl_id);
*
*/
H5_DLL hid_t H5Fcreate(const char *filename, unsigned flags, hid_t fcpl_id, hid_t fapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Fcreate}
*/
H5_DLL hid_t H5Fcreate_async(const char *app_file, const char *app_func, unsigned app_line,
const char *filename, unsigned flags, hid_t fcpl_id, hid_t fapl_id, hid_t es_id);
/**
@@ -408,6 +406,9 @@ H5_DLL hid_t H5Fcreate_async(const char *app_file, const char *app_func, unsigne
* identifier should be closed by calling H5Fclose() when it is no
* longer needed.
*
* \par Example
* \snippet H5F_examples.c open
*
* \note #H5F_ACC_RDWR and #H5F_ACC_RDONLY are mutually exclusive; use
* exactly one.
*
@@ -451,6 +452,11 @@ H5_DLL hid_t H5Fcreate_async(const char *app_file, const char *app_func, unsigne
*
*/
H5_DLL hid_t H5Fopen(const char *filename, unsigned flags, hid_t fapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Fopen}
*/
H5_DLL hid_t H5Fopen_async(const char *app_file, const char *app_func, unsigned app_line,
const char *filename, unsigned flags, hid_t access_plist, hid_t es_id);
/**
@@ -479,6 +485,11 @@ H5_DLL hid_t H5Fopen_async(const char *app_file, const char *app_func, unsigned
*
*/
H5_DLL hid_t H5Freopen(hid_t file_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Freopen}
*/
H5_DLL hid_t H5Freopen_async(const char *app_file, const char *app_func, unsigned app_line, hid_t file_id,
hid_t es_id);
/**
@@ -503,6 +514,9 @@ H5_DLL hid_t H5Freopen_async(const char *app_file, const char *app_func, unsigne
* global or local. Valid values are as follows:
* \scopes
*
* \par Example
* \snippet H5F_examples.c flush
*
* \attention HDF5 does not possess full control over buffering. H5Fflush()
* flushes the internal HDF5 buffers then asks the operating system
* (the OS) to flush the system buffers for the open files. After
@@ -511,13 +525,13 @@ H5_DLL hid_t H5Freopen_async(const char *app_file, const char *app_func, unsigne
*
*/
H5_DLL herr_t H5Fflush(hid_t object_id, H5F_scope_t scope);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Fflush}
*/
H5_DLL herr_t H5Fflush_async(const char *app_file, const char *app_func, unsigned app_line, hid_t object_id,
H5F_scope_t scope, hid_t es_id);
/**
* \example H5Fclose.c
* After creating an HDF5 file with H5Fcreate(), we close it with
* H5Fclose().
*/
/**
* \ingroup H5F
*
@@ -534,8 +548,8 @@ H5_DLL herr_t H5Fflush_async(const char *app_file, const char *app_func, unsigne
* identifier, or shared datatype identifier), the file will be fully
* closed and access will end.
*
* Use H5Fclose() as shown in the following example:
* \include H5Fclose.c
* \par Example
* \snippet H5F_examples.c minimal
*
* \note \Bold{Delayed close:} Note the following deviation from the
* above-described behavior. If H5Fclose() is called for a file but one
@@ -562,6 +576,11 @@ H5_DLL herr_t H5Fflush_async(const char *app_file, const char *app_func, unsigne
*
*/
H5_DLL herr_t H5Fclose(hid_t file_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Fclose}
*/
H5_DLL herr_t H5Fclose_async(const char *app_file, const char *app_func, unsigned app_line, hid_t file_id,
hid_t es_id);
/**
@@ -655,7 +674,7 @@ H5_DLL hid_t H5Fget_access_plist(hid_t file_id);
* \note The function will not return an error if intent is NULL; it will
* simply do nothing.
*
* \version 1.10.0 C function enhanced to work with SWMR functionality.
* \version 1.10.0 Function enhanced to work with SWMR functionality.
*
* \since 1.8.0
*
@@ -707,7 +726,7 @@ H5_DLL herr_t H5Fget_fileno(hid_t file_id, unsigned long *fileno);
* \c (#H5F_OBJ_DATASET|#H5F_OBJ_GROUP) would call for datasets and
* groups.
*
* \version 1.6.8, 1.8.2 C function return type changed to \c ssize_t.
* \version 1.6.8, 1.8.2 Function return type changed to \c ssize_t.
* \version 1.6.5 #H5F_OBJ_LOCAL has been added as a qualifier on the types
* of objects to be counted. #H5F_OBJ_LOCAL restricts the
* search to objects opened through current file identifier.
@@ -745,9 +764,9 @@ H5_DLL ssize_t H5Fget_obj_count(hid_t file_id, unsigned types);
* To retrieve a count of open objects, use the H5Fget_obj_count()
* function. This count can be used to set the \p max_objs parameter.
*
* \version 1.8.2 C function return type changed to \c ssize_t and \p
* \version 1.8.2 Function return type changed to \c ssize_t and \p
* max_objs parameter datatype changed to \c size_t.
* \version 1.6.8 C function return type changed to \c ssize_t and \p
* \version 1.6.8 Function return type changed to \c ssize_t and \p
* max_objs parameter datatype changed to \c size_t.
* \since 1.6.0
*
@@ -798,6 +817,9 @@ H5_DLL herr_t H5Fget_vfd_handle(hid_t file_id, hid_t fapl, void **file_handle);
* attribute, then the file will be mounted at the location where the
* attribute, dataset, or named datatype is attached.
*
* \par Example
* \snippet H5F_examples.c mount
*
* \note To date, no file mount properties have been defined in HDF5. The
* proper value to pass for \p plist is #H5P_DEFAULT, indicating the
* default file mount property list.
@@ -868,8 +890,6 @@ H5_DLL hssize_t H5Fget_freespace(hid_t file_id);
* if any, the HDF5 portion of the file, and any data that may have
* been appended beyond the data written through the HDF5 library.
*
* \version 1.6.3 Fortran subroutine introduced in this release.
*
* \since 1.6.3
*
*/
@@ -948,9 +968,7 @@ H5_DLL herr_t H5Fincrement_filesize(hid_t file_id, hsize_t increment);
*
* \note \Bold{Recommended Reading:} This function is part of the file image
* operations feature set. It is highly recommended to study the guide
* "HDF5 File Image Operations" before using this feature set.\n See the
* "See Also" section below for links to other elements of HDF5 file
* image operations. \todo Fix the references.
* \ref_file_image_ops before using this feature set.
*
* \attention H5Pget_file_image() will fail, returning a negative value, if the
* file is too large for the supplied buffer.
@@ -958,8 +976,6 @@ H5_DLL herr_t H5Fincrement_filesize(hid_t file_id, hsize_t increment);
* \see H5LTopen_file_image(), H5Pset_file_image(), H5Pget_file_image(),
* H5Pset_file_image_callbacks(), H5Pget_file_image_callbacks()
*
* \version 1.8.13 Fortran subroutine added in this release.
*
* \since 1.8.0
*
*/
@@ -976,197 +992,18 @@ H5_DLL ssize_t H5Fget_file_image(hid_t file_id, void *buf_ptr, size_t buf_len);
* \ref H5AC-cache-config-t "here".
* \return \herr_t
*
* \note The \c in direction applies only to the H5AC_cache_config_t::version
* field. All other fields are out parameters.
*
* \details H5Fget_mdc_config() loads the current metadata cache configuration
* into the instance of H5AC_cache_config_t pointed to by the \p config_ptr
* parameter.
*
* Note that the \c version field of \p config_ptr must be initialized
* --this allows the library to support old versions of the H5AC_cache_config_t
* structure.
*
* \par General configuration section
* <table>
* <tr>
* <td><em>int</em> <code>version</code> </td>
* <td>IN: Integer field indicating the the version of the H5AC_cache_config_t in use. This field should
* be set to #H5AC__CURR_CACHE_CONFIG_VERSION (defined in H5ACpublic.h).</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>rpt_fcn_enabled</code> </td>
* <td><p>OUT: Boolean flag indicating whether the adaptive cache resize report function is enabled. This
* field should almost always be set to disabled (<code>0</code>). Since resize algorithm activity is
* reported via stdout, it MUST be set to disabled (<code>0</code>) on Windows machines.</p><p>The
* report function is not supported code, and can be expected to change between versions of the
* library. Use it at your own risk.</p></td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>open_trace_file</code> </td>
* <td>OUT: Boolean field indicating whether the <code>trace_file_name</code> field should be used to
* open a trace file for the cache. This field will always be set to <code>0</code> in this
* context.</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>close_trace_file</code> </td>
* <td>OUT: Boolean field indicating whether the current trace file (if any) should be closed. This field
* will always be set to <code>0</code> in this context.</td></tr>
* <tr>
* <td><em>char*</em><code>trace_file_name</code> </td>
* <td>OUT: Full path name of the trace file to be opened if the <code>open_trace_file</code> field is
* set to <code>1</code>. This field will always be set to the empty string in this context.</td></tr>
* <tr>
* <td><em>hbool_t</em> <code>evictions_enabled</code> </td>
* <td>OUT: Boolean flag indicating whether metadata cache entry evictions are
* enabled.</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>set_initial_size</code> </td>
* <td>OUT: Boolean flag indicating whether the cache should be created with a user specified initial
* maximum size.<p>If the configuration is loaded from the cache, this flag will always be set
* to <code>0</code>.</p></td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>initial_size</code> </td>
* <td>OUT: Initial maximum size of the cache in bytes, if applicable.<p>If the configuration is loaded
* from the cache, this field will contain the cache maximum size as of the time of the
* call.</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>min_clean_fraction</code> </td>
* <td>OUT: Float value specifying the minimum fraction of the cache that must be kept either clean or
* empty when possible.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>max_size</code> </td>
* <td>OUT: Upper bound (in bytes) on the range of values that the adaptive cache resize code can select
* as the maximum cache size.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>min_size</code> </td>
* <td>OUT: Lower bound (in bytes) on the range of values that the adaptive cache resize code can select
* as the maximum cache size.</td>
* </tr>
* <tr>
* <td><em>long int</em> <code>epoch_length</code> </td>
* <td>OUT: Number of cache accesses between runs of the adaptive cache resize
* code.</td>
* </tr>
* </table>
*
* \par Increment configuration section
* <table>
* <tr>
* <td><em>enum H5C_cache_incr_mode</em> <code>incr_mode</code> </td>
* <td>OUT: Enumerated value indicating the operational mode of the automatic cache size increase code.
* At present, only the following values are legal:<p>\c H5C_incr__off: Automatic cache size increase
* is disabled.</p><p>\c H5C_incr__threshold: Automatic cache size increase is enabled using the hit
* rate threshold algorithm.</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>lower_hr_threshold</code> </td>
* <td>OUT: Hit rate threshold used in the hit rate threshold cache size increase algorithm.</td>
* </tr>
* <tr>
* <td><em>double</em> <code>increment</code> </td>
* <td>OUT: The factor by which the current maximum cache size is multiplied to obtain an initial new
* maximum cache size if a size increase is triggered in the hit rate threshold cache size increase
* algorithm.</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>apply_max_increment</code> </td>
* <td>OUT: Boolean flag indicating whether an upper limit will be applied to the size of cache size
* increases.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>max_increment</code> </td>
* <td>OUT: The maximum number of bytes by which the maximum cache size can be increased in a single step
* -- if applicable.</td>
* </tr>
* <tr>
* <td><em>enum H5C_cache_flash_incr_mode</em> <code>flash_incr_mode</code> </td>
* <td>OUT: Enumerated value indicating the operational mode of the flash cache size increase code. At
* present, only the following values are legal:<p>\c H5C_flash_incr__off: Flash cache size increase is
* disabled.</p><p>\c H5C_flash_incr__add_space: Flash cache size increase is enabled using the add
* space algorithm.</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>flash_threshold</code> </td>
* <td>OUT: The factor by which the current maximum cache size is multiplied to obtain the minimum size
* entry / entry size increase which may trigger a flash cache size
* increase.</td>
* </tr>
* <tr>
* <td><em>double</em> <code>flash_multiple</code> </td>
* <td>OUT: The factor by which the size of the triggering entry / entry size increase is multiplied to
* obtain the initial cache size increment. This increment may be reduced to reflect existing free
* space in the cache and the <code>max_size</code> field above.</td>
* </tr>
* </table>
*
* \par Decrement configuration section
* <table>
* <tr><td colspan="2"><strong>Decrement configuration
* section:</strong></td>
* </tr>
* <tr>
* <td><em>enum H5C_cache_decr_mode</em> <code>decr_mode</code> </td>
* <td>OUT: Enumerated value indicating the operational mode of the automatic cache size decrease code.
* At present, the following values are legal:<p>H5C_decr__off: Automatic cache size decrease is
* disabled, and the remaining decrement fields are ignored.</p><p>H5C_decr__threshold: Automatic
* cache size decrease is enabled using the hit rate threshold algorithm.</p><p>H5C_decr__age_out:
* Automatic cache size decrease is enabled using the ageout algorithm.</p>
* <p>H5C_decr__age_out_with_threshold: Automatic cache size decrease is enabled using the ageout
* with hit rate threshold algorithm</p></td>
* </tr>
* <tr><td><em>double</em> <code>upper_hr_threshold</code> </td>
* <td>OUT: Upper hit rate threshold. This value is only used if the decr_mode is either
* H5C_decr__threshold or H5C_decr__age_out_with_threshold.</td>
* </tr>
* <tr>
* <td><em>double</em> <code>decrement</code> </td>
* <td>OUT: Factor by which the current max cache size is multiplied to obtain an initial value for the
* new cache size when cache size reduction is triggered in the hit rate threshold cache size reduction
* algorithm.</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>apply_max_decrement</code> </td>
* <td>OUT: Boolean flag indicating whether an upper limit should be applied to the size of cache size
* decreases.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>max_decrement</code> </td>
* <td>OUT: The maximum number of bytes by which cache size can be decreased if any single step, if
* applicable.</td>
* </tr>
* <tr>
* <td><em>int</em> <code>epochs_before_eviction</code> </td>
* <td>OUT: The minimum number of epochs that an entry must reside unaccessed in cache before being
* evicted under either of the ageout cache size reduction algorithms.</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>apply_empty_reserve</code> </td>
* <td>OUT: Boolean flag indicating whether an empty reserve should be maintained under either of the
* ageout cache size reduction algorithms.</td>
* </tr>
* <tr>
* <td><em>double</em> <code>empty_reserve</code> </td>
* <td>OUT: Empty reserve for use with the ageout cache size reduction algorithms, if applicable.</td>
* </tr>
* </table>
*
* \par Parallel configuration section
* <table>
* <tr><td><em>int</em> <code>dirty_bytes_threshold</code> </td>
* <td>OUT: Threshold number of bytes of dirty metadata generation for triggering synchronizations of the
* metadata caches serving the target file in the parallel case.<p>Synchronization occurs whenever the
* number of bytes of dirty metadata created since the last synchronization exceeds this
* limit.</p></td>
* </tr>
* </table>
* parameter.\n
* The fields of the H5AC_cache_config_t structure are shown below:
* \snippet H5ACpublic.h H5AC_cache_config_t_snip
* \click4more
*
* \since 1.8.0
*
* \todo Fix the reference!
*
*/
H5_DLL herr_t H5Fget_mdc_config(hid_t file_id, H5AC_cache_config_t *config_ptr);
/**
@@ -1183,240 +1020,11 @@ H5_DLL herr_t H5Fget_mdc_config(hid_t file_id, H5AC_cache_config_t *config_ptr);
*
* \details H5Fset_mdc_config() attempts to configure the file's metadata cache
* according configuration supplied in \p config_ptr.
*
* \par General configuration fields
* <table>
* <tr>
* <td><em>int</em> <code>version</code></td>
* <td>IN: Integer field indicating the the version of the H5AC_cache_config_t in use. This
* field should be set to #H5AC__CURR_CACHE_CONFIG_VERSION (defined
* in H5ACpublic.h).</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>rpt_fcn_enabled</code></td>
* <td>IN: Boolean flag indicating whether the adaptive cache resize report function is enabled. This
* field should almost always be set to disabled (<code>0</code>). Since resize algorithm activity is
* reported via stdout, it MUST be set to disabled (<code>0</code>) on Windows machines.<p>The report
* function is not supported code, and can be expected to change between versions of the library. Use
* it at your own risk.</p></td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>open_trace_File</code></td>
* <td>IN: Boolean field indicating whether the <code>trace_file_name</code> field should be used to open
* a trace file for the cache.<p>The trace file is a debuging feature that allows the capture of top
* level metadata cache requests for purposes of debugging and/or optimization. This field should
* <p>This field should only normally be set to <code>0</code>, as trace file collection imposes
* considerable overhead.</p> be set to <code>1</code> when the <code>trace_file_name</code> contains
* the full path of the desired trace file, and either there is no open trace file on the cache, or the
* <code>close_trace_file</code> field is also <code>1</code>.</p><p>The trace file feature is
* unsupported unless used at the direction of The HDF Group. It is intended to allow The HDF Group to
* collect a trace of cache activity in cases of occult failures and/or poor performance seen in the
* field, so as to aid in reproduction in the lab. If you use it absent the direction of The HDF Group,
* you are on your own.</p></td>
* </tr>
* <tr><td><em>hbool_t</em> <code>close_trace_file</code></td>
* <td>IN: Boolean field indicating whether the current trace file (if any) should be closed.<p>See the
* above comments on the <code>open_trace_file</code> field. This field should be set to
* <code>0</code> unless there is an open trace file on the cache that you wish to close.</p><p>The
* trace file feature is unsupported unless used at the direction of The HDF Group. It is intended to
* allow The HDF Group to collect a trace of cache activity in cases of occult failures and/or poor
* performance seen in the field, so as to aid in reproduction in the lab. If you use it absent the
* direction of The HDF Group, you are on your own.</p></td>
* </tr>
* <tr>
* <td><em>char</em> <code>trace_file_name[]</code></td>
* <td>IN: Full path of the trace file to be opened if the <code>open_trace_file</code> field is set
* to <code>1</code>.<p>In the parallel case, an ascii representation of the mpi rank of the process
* will be appended to the file name to yield a unique trace file name for each process.</p><p>The
* length of the path must not exceed #H5AC__MAX_TRACE_FILE_NAME_LEN characters.</p><p>The trace file
* feature is unsupported unless used at the direction of The HDF Group. It is intended to allow The
* HDF Group to collect a trace of cache activity in cases of occult failures and/or poor performance
* seen in the field, so as to aid in reproduction in the lab. If you use it absent the direction of
* The HDF Group, you are on your own.</p></td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>evictions_enabled</code></td>
* <td>IN: A boolean flag indicating whether evictions from the metadata cache are enabled. This flag is
* initially set to enabled (<code>1</code>).<p>In rare circumstances, the raw data throughput
* requirements may be so high that the user wishes to postpone metadata writes so as to reserve I/O
* throughput for raw data. The <code>evictions_enabled</code> field exists to allow this. However,
* this is an extreme step, and you have no business doing it unless you have read the User Guide
* section on metadata caching, and have considered all other options carefully.</p><p>The
* <code>evictions_enabled</code> field may not be set to disabled (<code>0</code>) unless all adaptive
* cache resizing code is disabled via the <code>incr_mode</code>, <code>flash_incr_mode</code>, and
* <code>decr_mode</code> fields.</p><p>When this flag is set to disabled (<code>0</code>), the
* metadata cache will not attempt to evict entries to make space for new entries, and thus will grow
* without bound.</p><p>Evictions will be re-enabled when this field is set back to <code>1</code>.
* This should be done as soon as possible.</p></td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>set_initial_size</code></td>
* <td>IN: Boolean flag indicating whether the cache should be forced to the user specified initial
* size.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>initial_size</code></td>
* <td>IN: If <code>set_initial_size</code> is set to <code>1</code>, then <code>initial_size</code> must
* contain the desired initial size in bytes. This value must lie in the closed interval
* <code>[min_size, max_size]</code>. (see below)</td>
* </tr>
* <tr><td><em>double</em> <code>min_clean_fraction</code></td>
* <td>IN: This field specifies the minimum fraction of the cache that must be kept either clean or
* empty.<p>The value must lie in the interval [0.0, 1.0]. 0.01 is a good place to start in the serial
* case. In the parallel case, a larger value is needed -- see <a
* href="/display/HDF5/Metadata+Caching+in+HDF5">Metadata Caching in HDF5</a> in the collection
* "Advanced Topics in HDF5."</p></td>
* </tr>
* <tr><td><em>size_t</em> <code>max_size</code></td>
* <td>IN: Upper bound (in bytes) on the range of values that the adaptive cache resize code can select
* as the maximum cache size.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>min_size</code></td>
* <td>IN: Lower bound (in bytes) on the range of values that the adaptive cache resize code can select
* as the maximum cache size.</td>
* </tr>
* <tr><td><em>long int</em> <code>epoch_length</code></td>
* <td>IN: Number of cache accesses between runs of the adaptive cache resize code. 50,000 is a good
* starting number.</td>
* </tr>
* </table>
*
* \par Increment configuration fields
* <table>
* <tr>
* <td><em>enum H5C_cache_incr_mode</em> <code>incr_mode</code></td>
* <td>IN: Enumerated value indicating the operational mode of the automatic cache size increase code. At
* present, only two values are legal:<p>\c H5C_incr__off: Automatic cache size increase is disabled,
* and the remaining increment fields are ignored.</p><p>\c H5C_incr__threshold: Automatic cache size
* increase is enabled using the hit rate threshold algorithm.</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>lower_hr_threshold</code></td>
* <td>IN: Hit rate threshold used by the hit rate threshold cache size increment algorithm.<p>When the
* hit rate over an epoch is below this threshold and the cache is full, the maximum size of the
* cache is multiplied by increment (below), and then clipped as necessary to stay within max_size, and
* possibly max_increment.</p><p>This field must lie in the interval [0.0, 1.0]. 0.8 or 0.9 is a good
* starting point.</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>increment</code></td>
* <td>IN: Factor by which the hit rate threshold cache size increment algorithm multiplies the current
* maximum cache size to obtain a tentative new cache size.<p>The actual cache size increase will be
* clipped to satisfy the max_size specified in the general configuration, and possibly max_increment
* below.</p><p>The parameter must be greater than or equal to 1.0 -- 2.0 is a reasonable
* value.</p><p>If you set it to 1.0, you will effectively disable cache size increases.</p></td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>apply_max_increment</code></td>
* <td>IN: Boolean flag indicating whether an upper limit should be applied to the size of cache size
* increases.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>max_increment</code></td>
* <td>IN: Maximum number of bytes by which cache size can be increased in a single step -- if
* applicable.</td>
* </tr>
* <tr>
* <td><em>enum H5C_cache_flash_incr_mode</em> <code>flash_incr_mode</code></td>
* <td>IN: Enumerated value indicating the operational mode of the flash cache size increase code. At
* present, only the following values are legal:<p>\c H5C_flash_incr__off: Flash cache size increase is
* disabled.</p><p>\c H5C_flash_incr__add_space: Flash cache size increase is enabled using the add
* space algorithm.</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>flash_threshold</code></td>
* <td>IN: The factor by which the current maximum cache size is multiplied to obtain the minimum size
* entry / entry size increase which may trigger a flash cache size increase.<p>At present, this value
* must lie in the range [0.1, 1.0].</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>flash_multiple</code></td>
* <td>IN: The factor by which the size of the triggering entry / entry size increase is multiplied to
* obtain the initial cache size increment. This increment may be reduced to reflect existing free
* space in the cache and the <code>max_size</code> field above.<p>At present, this field must lie in
* the range [0.1, 10.0].</p></td>
* </tr>
* </table>
*
* \par Decrement configuration fields
* <table>
* <tr>
* <td><em>enum H5C_cache_decr_mode</em> <code>decr_mode</code></td>
* <td>IN: Enumerated value indicating the operational mode of the automatic cache size decrease code. At
* present, the following values are legal:<p>\c H5C_decr__off: Automatic cache size decrease is
* disabled.</p><p>\c H5C_decr__threshold: Automatic cache size decrease is enabled using the hit
* rate threshold algorithm.</p><p>\c H5C_decr__age_out: Automatic cache size decrease is enabled using
* the ageout algorithm.</p><p>\c H5C_decr__age_out_with_threshold: Automatic cache size decrease is
* enabled using the ageout with hit rate threshold algorithm</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>upper_hr_threshold</code></td>
* <td>IN: Hit rate threshold for the hit rate threshold and ageout with hit rate threshold cache size
* decrement algorithms.<p>When \c decr_mode is \c H5C_decr__threshold, and the hit rate over a given
* epoch exceeds the supplied threshold, the current maximum cache size is multiplied by decrement to
* obtain a tentative new (and smaller) maximum cache size.</p><p>When \c decr_mode is \c
* H5C_decr__age_out_with_threshold, there is no attempt to find and evict aged out entries unless the
* hit rate in the previous epoch exceeded the supplied threshold.</p><p>This field must lie in the
* interval [0.0, 1.0].</p><p>For \c H5C_incr__threshold, .9995 or .99995 is a good place to
* start.</p><p>For \c H5C_decr__age_out_with_threshold, .999 might be more useful.</p></td>
* </tr>
* <tr>
* <td><em>double</em> <code>decrement</code></td>
* <td>IN: In the hit rate threshold cache size decrease algorithm, this parameter contains the factor by
* which the current max cache size is multiplied to produce a tentative new cache size.<p>The actual
* cache size decrease will be clipped to satisfy the min_size specified in the general configuration,
* and possibly max_decrement below.</p><p>The parameter must be be in the interval
* [0.0, 1.0].</p><p>If you set it to 1.0, you will effectively disable cache size decreases. 0.9 is a
* reasonable starting point.</p></td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>apply_max_decrement</code></td>
* <td>IN: Boolean flag indicating whether an upper limit should be applied to the size of cache size
* decreases.</td>
* </tr>
* <tr>
* <td><em>size_t</em> <code>max_decrement</code></td>
* <td>IN: Maximum number of bytes by which the maximum cache size can be decreased in any single step --
* if applicable.</td>
* </tr>
* <tr>
* <td><em>int</em> <code>epochs_before_eviction</code></td>
* <td>IN: In the ageout based cache size reduction algorithms, this field contains the minimum number of
* epochs an entry must remain unaccessed in cache before the cache size reduction algorithm tries to
* evict it. 3 is a reasonable value.</td>
* </tr>
* <tr>
* <td><em>hbool_t</em> <code>apply_empty_reserve</code></td>
* <td>IN: Boolean flag indicating whether the ageout based decrement algorithms will maintain a empty
* reserve when decreasing cache size.</td>
* </tr>
* <tr>
* <td><em>double</em> <code>empty_reserve</code></td>
* <td>IN: Empty reserve as a fraction of maximum cache size if applicable.<p>When so directed, the
* ageout based algorithms will not decrease the maximum cache size unless the empty reserve can be
* met.</p><p>The parameter must lie in the interval [0.0, 1.0]. 0.1 or 0.05 is a good place to
* start.</p></td>
* </tr>
* </table>
*
* \par Parallel configuration fields
* <table>
* <tr>
* <td><em>int</em> <code>dirty_bytes_threshold</code></td>
* <td>IN: Threshold number of bytes of dirty metadata generation for triggering synchronizations of the
* metadata caches serving the target file in the parallel case.<p>Synchronization occurs whenever the
* number of bytes of dirty metadata created since the last synchronization exceeds this
* limit.</p><p>This field only applies to the parallel case. While it is ignored elsewhere, it can
* still draw a value out of bounds error.</p><p>It must be consistant across all caches on any given
* file.</p><p>By default, this field is set to 256 KB. It shouldn't be more than half the current
* maximum cache size times the minimum clean fraction.</p></td>
* </tr>
* </table>
* \snippet H5ACpublic.h H5AC_cache_config_t_snip
* \click4more
*
* \since 1.8.0
*
* \todo Fix the MDC document reference!
*/
H5_DLL herr_t H5Fset_mdc_config(hid_t file_id, H5AC_cache_config_t *config_ptr);
/**
@@ -1495,13 +1103,12 @@ H5_DLL herr_t H5Fget_mdc_size(hid_t file_id, size_t *max_size_ptr, size_t *min_c
* is enabled. However, the call should be useful if you choose to control metadata cache size from your
* program.
*
* See "Metadata Caching in HDF5" for details about the metadata cache and the adaptive cache resizing
* See \ref_mdc_in_hdf5 for details about the metadata cache and the adaptive cache resizing
* algorithms. If you have not read, understood, and thought about the material covered in that
* documentation,
* you should not be using this API call.
* \endparblock
*
* \todo Fix the MDC document reference!
*/
H5_DLL herr_t H5Freset_mdc_hit_rate_stats(hid_t file_id);
/**
@@ -1797,6 +1404,9 @@ H5_DLL herr_t H5Fclear_elink_file_cache(hid_t file_id);
* For the parameters \p low and \p high, see the description for
* H5Pset_libver_bounds().
*
* \par Example
* \snippet H5F_examples.c libver_bounds
*
* \since 1.10.2
*
*/
@@ -1835,7 +1445,7 @@ H5_DLL herr_t H5Fset_libver_bounds(hid_t file_id, H5F_libver_t low, H5F_libver_t
* list, and H5Fget_mdc_logging_status() will return the current state of
* the logging flags.
*
* The log format is described in the \Emph{Metadata Cache Logging} document.
* The log format is described in the \ref_mdc_logging document.
*
* \note Logging can only be started or stopped if metadata cache logging was enabled
* via H5Pset_mdc_log_options().\n
@@ -1849,8 +1459,6 @@ H5_DLL herr_t H5Fset_libver_bounds(hid_t file_id, H5F_libver_t low, H5F_libver_t
*
* \since 1.10.0
*
* \todo Fix the document reference!
*
*/
H5_DLL herr_t H5Fstart_mdc_logging(hid_t file_id);
/**
@@ -1887,7 +1495,7 @@ H5_DLL herr_t H5Fstart_mdc_logging(hid_t file_id);
* list, and H5Fget_mdc_logging_status() will return the current state of
* the logging flags.
*
* The log format is described in the \Emph{Metadata Cache Logging} document.
* The log format is described in the \ref_mdc_logging document.
*
* \note Logging can only be started or stopped if metadata cache logging was enabled
* via H5Pset_mdc_log_options().\n
@@ -1933,7 +1541,7 @@ H5_DLL herr_t H5Fstop_mdc_logging(hid_t file_id);
* list, and H5Fget_mdc_logging_status() will return the current state of
* the logging flags.
*
* The log format is described in the \Emph{Metadata Cache Logging} document.
* The log format is described in the \ref_mdc_logging document.
*
* \note Unlike H5Fstart_mdc_logging() and H5Fstop_mdc_logging(), this function can
* be called on any open file identifier.
@@ -1944,7 +1552,7 @@ H5_DLL herr_t H5Fget_mdc_logging_status(hid_t file_id, hbool_t *is_enabled, hboo
/**
* \ingroup SWMR
*
* \todo Finish this!
* \todo UFO?
*/
H5_DLL herr_t H5Fformat_convert(hid_t fid);
/**
@@ -1998,7 +1606,7 @@ H5_DLL herr_t H5Fget_page_buffering_stats(hid_t file_id, unsigned accesses[2], u
* \brief Obtains information about a cache image if it exists
*
* \file_id
* \param[out] image_addr Offset of the cache image if it exists, or #HADDR_UNDEF if it does not
* \param[out] image_addr Offset of the cache image if it exists, or \c HADDR_UNDEF if it does not
* \param[out] image_size Length of the cache image if it exists, or 0 if it does not
* \returns \herr_t
*
@@ -2139,11 +1747,10 @@ H5_DLL herr_t H5Fwait(hid_t file_id);
* the desired behavior.
* \endparblock
*
* \see Enabling a Strict Consistency Semantics Model in Parallel HDF5
* \see \ref_cons_semantics
*
* \since 1.8.9
*
* \todo Fix the reference!
*/
H5_DLL herr_t H5Fset_mpi_atomicity(hid_t file_id, hbool_t flag);
/**
@@ -2163,11 +1770,10 @@ H5_DLL herr_t H5Fset_mpi_atomicity(hid_t file_id, hbool_t flag);
* Upon successful return, \p flag will be set to \c 1 if file access is set
* to atomic mode and \c 0 if file access is set to nonatomic mode.
*
* \see Enabling a Strict Consistency Semantics Model in Parallel HDF5
* \see \ref_cons_semantics
*
* \since 1.8.9
*
* \todo Fix the reference!
*/
H5_DLL herr_t H5Fget_mpi_atomicity(hid_t file_id, hbool_t *flag);
#endif /* H5_HAVE_PARALLEL */
@@ -2199,14 +1805,14 @@ H5_DLL herr_t H5Fget_mpi_atomicity(hid_t file_id, hbool_t *flag);
#ifndef H5_NO_DEPRECATED_SYMBOLS
/* Macros */
#define H5F_ACC_DEBUG (H5CHECK H5OPEN 0x0000u) /*print debug info (deprecated)*/
#define H5F_ACC_DEBUG (H5CHECK H5OPEN 0x0000u) /**< Print debug info \deprecated In which version? */
/* Typedefs */
/**
* Current "global" information about file
*/
//! [H5F_info1_t_snip]
//! <!-- [H5F_info1_t_snip] -->
typedef struct H5F_info1_t {
hsize_t super_ext_size; /**< Superblock extension size */
struct {
@@ -2214,7 +1820,7 @@ typedef struct H5F_info1_t {
H5_ih_info_t msgs_info; /**< Shared object header message index & heap size */
} sohm;
} H5F_info1_t;
//! [H5F_info1_t_snip]
//! <!-- [H5F_info1_t_snip] -->
/* Function prototypes */
/**
@@ -2252,7 +1858,7 @@ typedef struct H5F_info1_t {
* header indexes. Each index might be either a B-tree or
* a list.
*
* \version 1.10.0 C function H5Fget_info() renamed to H5Fget_info1() and
* \version 1.10.0 Function H5Fget_info() renamed to H5Fget_info1() and
* deprecated in this release.
*
* \since 1.8.0
src/H5Gmodule.h
+88 -3
View File
@@ -31,9 +31,94 @@
/**
* \defgroup H5G H5G
* \brief Group Interface
* \details The HDF5 Group Interface, H5G, provides a mechanism for managing
* HDF5 groups and their members, which are other HDF5 objects.
*
* \details \Bold{Groups in HDF5:} A group associates names with objects and
* provides a mechanism for mapping a name to an object. Since all
* objects appear in at least one group (with the possible exception of
* the root object) and since objects can have names in more than one
* group, the set of all objects in an HDF5 file is a directed
* graph. The internal nodes (nodes with out-degree greater than zero)
* must be groups while the leaf nodes (nodes with out-degree zero) are
* either empty groups or objects of some other type. Exactly one
* object in every non-empty file is the root object. The root object
* always has a positive in-degree because it is pointed to by the file
* super block.
*
* \Bold{Locating objects in the HDF5 file hierarchy:} An object name
* consists of one or more components separated from one another by
* slashes. An absolute name begins with a slash and the object is
* located by looking for the first component in the root object, then
* looking for the second component in the first object, etc., until
* the entire name is traversed. A relative name does not begin with a
* slash and the traversal begins at the location specified by the
* create or access function.
*
* \Bold{Group implementations in HDF5:} The original HDF5 group
* implementation provided a single indexed structure for link
* storage. A new group implementation, in HDF5 Release 1.8.0, enables
* more efficient compact storage for very small groups, improved link
* indexing for large groups, and other advanced features.
*
* \li The \Emph{original indexed} format remains the default. Links
* are stored in a B-tree in the group’s local heap.
* \li Groups created in the new \Emph{compact-or-indexed} format, the
* implementation introduced with Release 1.8.0, can be tuned for
* performance, switching between the compact and indexed formats
* at thresholds set in the user application.
* - The \Emph{compact} format will conserve file space and processing
* overhead when working with small groups and is particularly
* valuable when a group contains no links. Links are stored
* as a list of messages in the group’s header.
* - The \Emph{indexed} format will yield improved
* performance when working with large groups, e.g., groups
* containing thousands to millions of members. Links are stored in
* a fractal heap and indexed with an improved B-tree.
* \li The new implementation also enables the use of link names consisting of
* non-ASCII character sets (see H5Pset_char_encoding()) and is
* required for all link types other than hard or soft links, e.g.,
* external and user-defined links (see the \ref H5L APIs).
*
* The original group structure and the newer structures are not
* directly interoperable. By default, a group will be created in the
* original indexed format. An existing group can be changed to a
* compact-or-indexed format if the need arises; there is no capability
* to change back. As stated above, once in the compact-or-indexed
* format, a group can switch between compact and indexed as needed.
*
* Groups will be initially created in the compact-or-indexed format
* only when one or more of the following conditions is met:
* \li The low version bound value of the library version bounds property
* has been set to Release 1.8.0 or later in the file access property
* list (see H5Pset_libver_bounds()). Currently, that would require an
* H5Pset_libver_bounds() call with the low parameter set to
* #H5F_LIBVER_LATEST.\n When this property is set for an HDF5 file,
* all objects in the file will be created using the latest available
* format; no effort will be made to create a file that can be read by
* older libraries.
* \li The creation order tracking property, #H5P_CRT_ORDER_TRACKED, has been
* set in the group creation property list (see H5Pset_link_creation_order()).
*
* An existing group, currently in the original indexed format, will be
* converted to the compact-or-indexed format upon the occurrence of
* any of the following events:
* \li An external or user-defined link is inserted into the group.
* \li A link named with a string composed of non-ASCII characters is
* inserted into the group.
*
* The compact-or-indexed format offers performance improvements that
* will be most notable at the extremes, i.e., in groups with zero
* members and in groups with tens of thousands of members. But
* measurable differences may sometimes appear at a threshold as low as
* eight group members. Since these performance thresholds and criteria
* differ from application to application, tunable settings are
* provided to govern the switch between the compact and indexed
* formats (see H5Pset_link_phase_change()). Optimal thresholds will
* depend on the application and the operating environment.
*
* Future versions of HDF5 will retain the ability to create, read,
* write, and manipulate all groups stored in either the original
* indexed format or the compact-or-indexed format.
*
*/
#endif /* H5Gmodule_H */
src/H5Gpublic.h
+666 -160
View File
@@ -41,24 +41,31 @@
/* Public Typedefs */
/*******************/
/* Types of link storage for groups */
//! <!-- [H5G_storage_t_snip] -->
/**
* Types of link storage for groups
*/
typedef enum H5G_storage_type_t {
H5G_STORAGE_TYPE_UNKNOWN = -1, /* Unknown link storage type */
H5G_STORAGE_TYPE_SYMBOL_TABLE, /* Links in group are stored with a "symbol table" */
/* (this is sometimes called "old-style" groups) */
H5G_STORAGE_TYPE_COMPACT, /* Links are stored in object header */
H5G_STORAGE_TYPE_DENSE /* Links are stored in fractal heap & indexed with v2 B-tree */
H5G_STORAGE_TYPE_UNKNOWN = -1, /**< Unknown link storage type */
H5G_STORAGE_TYPE_SYMBOL_TABLE, /**< Links in group are stored with a "symbol table" */
/**< (this is sometimes called "old-style" groups) */
H5G_STORAGE_TYPE_COMPACT, /**< Links are stored in object header */
H5G_STORAGE_TYPE_DENSE /**< Links are stored in fractal heap & indexed with v2 B-tree */
} H5G_storage_type_t;
//! <!-- [H5G_storage_t_snip] -->
/* Information struct for group (for H5Gget_info/H5Gget_info_by_name/H5Gget_info_by_idx) */
//! [H5G_info_t_snip]
//! <!-- [H5G_info_t_snip] -->
/**
* Information struct for group for
* H5Gget_info(), H5Gget_info_by_name(), and H5Gget_info_by_idx()
*/
typedef struct H5G_info_t {
H5G_storage_type_t storage_type; /* Type of storage for links in group */
hsize_t nlinks; /* Number of links in group */
int64_t max_corder; /* Current max. creation order value for group */
hbool_t mounted; /* Whether group has a file mounted on it */
H5G_storage_type_t storage_type; /**< Type of storage for links in group */
hsize_t nlinks; /**< Number of links in group */
int64_t max_corder; /**< Current max. creation order value for group */
hbool_t mounted; /**< Whether group has a file mounted on it */
} H5G_info_t;
//! [H5G_info_t_snip]
//! <!-- [H5G_info_t_snip] -->
/********************/
/* Public Variables */
@@ -120,24 +127,8 @@ H5_DLL hid_t H5Gcreate2(hid_t loc_id, const char *name, hid_t lcpl_id, hid_t gcp
/**
* --------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Asynchronous version of H5Gcreate2()
*
* \app_file
* \app_func
* \app_line
* \fgdta_loc_id
* \param[in] name Name of the group to create
* \lcpl_id
* \gcpl_id
* \gapl_id
* \es_id
*
* \return \hid_t{group}
*
* \see H5Gcreate2()
*
* \ingroup ASYNC
* \async_variant_of{H5Gcreate}
*/
H5_DLL hid_t H5Gcreate_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t lcpl_id, hid_t gcpl_id, hid_t gapl_id, hid_t es_id);
@@ -223,22 +214,8 @@ H5_DLL hid_t H5Gopen2(hid_t loc_id, const char *name, hid_t gapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Asynchronous version of H5Gopen2()
*
* \app_file
* \app_func
* \app_line
* \fgdta_loc_id
* \param[in] name Name of the group to open
* \gapl_id
* \es_id
*
* \return \hid_t{group}
*
* \see H5Gopen2()
*
* \ingroup ASYNC
* \async_variant_of{H5Gopen}
*/
H5_DLL hid_t H5Gopen_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t gapl_id, hid_t es_id);
@@ -296,21 +273,8 @@ H5_DLL herr_t H5Gget_info(hid_t loc_id, H5G_info_t *ginfo);
/**
* --------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Asynchronous version of H5Gget_info()
*
* \app_file
* \app_func
* \app_line
* \fgdta_loc_id
* \param[out] ginfo Struct in which group information is returned
* \es_id
*
* \return \hid_t{group}
*
* \see H5Gget_info()
*
* \ingroup ASYNC
* \async_variant_of{H5Gget_info}
*/
H5_DLL herr_t H5Gget_info_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
H5G_info_t *ginfo /*out*/, hid_t es_id);
@@ -351,23 +315,8 @@ H5_DLL herr_t H5Gget_info_by_name(hid_t loc_id, const char *name, H5G_info_t *gi
/**
* --------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Asynchronous version of H5Gget_info_by_name()
*
* \app_file
* \app_func
* \app_line
* \fgdta_loc_id
* \param[in] name Name of the group to query
* \param[out] ginfo Struct in which group information is returned
* \lapl_id
* \es_id
*
* \return \herr_t
*
* \see H5Gget_info_by_name()
*
* \ingroup ASYNC
* \async_variant_of{H5Gget_info_by_name}
*/
H5_DLL herr_t H5Gget_info_by_name_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *name, H5G_info_t *ginfo /*out*/,
@@ -423,29 +372,8 @@ H5_DLL herr_t H5Gget_info_by_idx(hid_t loc_id, const char *group_name, H5_index_
/**
* --------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Asynchronous version of H5Gcreate2()
*
* \app_file
* \app_func
* \app_line
* \fgdta_loc_id
* \param[in] group_name Name of the group to query
* \param[in] idx_type Transient index identifying object
* \param[in] order Transient index identifying object
* \param[in] n Position in the index of the group to query
* \param[out] ginfo Struct in which group information is returned
* \lapl_id
* \es_id
*
* \return Returns
* \li The size of the object name if successful, or
* \li 0 if no name is associated with the group identifier, or
* \li negative value, if failure occurred
*
* \see H5Gcreate2()
*
* \ingroup ASYNC
* \async_variant_of{H5Gget_info_by_idx}
*/
H5_DLL herr_t H5Gget_info_by_idx_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *group_name, H5_index_t idx_type,
@@ -523,29 +451,18 @@ H5_DLL herr_t H5Grefresh(hid_t group_id);
* Failure to release a group with this call will result in
* resource leaks.
*
* \since 1.0.0
* \par Example
* \snippet H5F_examples.c mount
*
* \version 1.4.0 Fortran function introduced in this release
* \since 1.0.0
*
*/
H5_DLL herr_t H5Gclose(hid_t group_id);
/**
* --------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Asynchronous version of H5Gcreate2()
*
* \app_file
* \app_func
* \app_line
* \group_id
* \es_id
*
* \return \herr_t
*
* \see H5Gcreate2()
*
* \ingroup ASYNC
* \async_variant_of{H5Gclose}
*/
H5_DLL herr_t H5Gclose_async(const char *app_file, const char *app_func, unsigned app_line, hid_t group_id,
hid_t es_id);
@@ -595,60 +512,649 @@ H5_DLL herr_t H5Gclose_async(const char *app_file, const char *app_func, unsigne
/* Typedefs */
/*
//! <!-- [H5G_obj_t_snip] -->
/**
* An object has a certain type. The first few numbers are reserved for use
* internally by HDF5. Users may add their own types with higher values. The
* values are never stored in the file -- they only exist while an
* application is running. An object may satisfy the `isa' function for more
* than one type.
* values are never stored in the file -- they only exist while an application
* is running. An object may satisfy the `isa' function for more than one type.
*
* \deprecated
*/
typedef enum H5G_obj_t {
H5G_UNKNOWN = -1, /* Unknown object type */
H5G_GROUP, /* Object is a group */
H5G_DATASET, /* Object is a dataset */
H5G_TYPE, /* Object is a named data type */
H5G_LINK, /* Object is a symbolic link */
H5G_UDLINK, /* Object is a user-defined link */
H5G_RESERVED_5, /* Reserved for future use */
H5G_RESERVED_6, /* Reserved for future use */
H5G_RESERVED_7 /* Reserved for future use */
H5G_UNKNOWN = -1, /**< Unknown object type */
H5G_GROUP, /**< Object is a group */
H5G_DATASET, /**< Object is a dataset */
H5G_TYPE, /**< Object is a named data type */
H5G_LINK, /**< Object is a symbolic link */
H5G_UDLINK, /**< Object is a user-defined link */
H5G_RESERVED_5, /**< Reserved for future use */
H5G_RESERVED_6, /**< Reserved for future use */
H5G_RESERVED_7 /**< Reserved for future use */
} H5G_obj_t;
//! <!-- [H5G_obj_t_snip] -->
/** Define the operator function pointer for for H5Giterate() */
//! [H5G_iterate_t_snip]
//! <!-- [H5G_iterate_t_snip] -->
/**
* Callback for H5Giterate()
*
* \deprecated
*/
typedef herr_t (*H5G_iterate_t)(hid_t group, const char *name, void *op_data);
//! [H5G_iterate_t_snip]
//! <!-- [H5G_iterate_t_snip] -->
/** Information about an object */
//! [H5G_stat_t_snip]
//! <!-- [H5G_stat_t_snip] -->
/**
* Information about an object
*
* \deprecated
*/
typedef struct H5G_stat_t {
unsigned long fileno[2]; /*file number */
unsigned long objno[2]; /*object number */
unsigned nlink; /*number of hard links to object*/
H5G_obj_t type; /*basic object type */
time_t mtime; /*modification time */
size_t linklen; /*symbolic link value length */
H5O_stat_t ohdr; /* Object header information */
unsigned long fileno[2]; /**< file number */
unsigned long objno[2]; /**< object number */
unsigned nlink; /**< number of hard links to object*/
H5G_obj_t type; /**< basic object type */
time_t mtime; /**< modification time */
size_t linklen; /**< symbolic link value length */
H5O_stat_t ohdr; /**< Object header information */
} H5G_stat_t;
//! [H5G_stat_t_snip]
//! <!-- [H5G_stat_t_snip] -->
/* Function prototypes */
H5_DLL hid_t H5Gcreate1(hid_t loc_id, const char *name, size_t size_hint);
H5_DLL hid_t H5Gopen1(hid_t loc_id, const char *name);
H5_DLL herr_t H5Glink(hid_t cur_loc_id, H5G_link_t type, const char *cur_name, const char *new_name);
H5_DLL herr_t H5Glink2(hid_t cur_loc_id, const char *cur_name, H5G_link_t type, hid_t new_loc_id,
const char *new_name);
H5_DLL herr_t H5Gmove(hid_t src_loc_id, const char *src_name, const char *dst_name);
H5_DLL herr_t H5Gmove2(hid_t src_loc_id, const char *src_name, hid_t dst_loc_id, const char *dst_name);
H5_DLL herr_t H5Gunlink(hid_t loc_id, const char *name);
H5_DLL herr_t H5Gget_linkval(hid_t loc_id, const char *name, size_t size, char *buf /*out*/);
H5_DLL herr_t H5Gset_comment(hid_t loc_id, const char *name, const char *comment);
H5_DLL int H5Gget_comment(hid_t loc_id, const char *name, size_t bufsize, char *buf);
H5_DLL herr_t H5Giterate(hid_t loc_id, const char *name, int *idx, H5G_iterate_t op, void *op_data);
H5_DLL herr_t H5Gget_num_objs(hid_t loc_id, hsize_t *num_objs);
H5_DLL herr_t H5Gget_objinfo(hid_t loc_id, const char *name, hbool_t follow_link,
H5G_stat_t *statbuf /*out*/);
H5_DLL ssize_t H5Gget_objname_by_idx(hid_t loc_id, hsize_t idx, char *name, size_t size);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Creates a new group and links it into the file
*
* \fgdta_loc_id
* \param[in] name Name of the group to create
* \param[in] size_hint Optional parameter indicating the number of bytes
* to reserve for the names that will appear in the group
*
* \return \hid_t{group}
*
* \deprecated This function is deprecated in favor of H5Gcreate2().
*
* \details H5Gcreate1() creates a new group with the specified name at the
* specified location, \p loc_id. \p loc_id may be a file, group,
* dataset, named datatype or attribute. If an attribute, dataset, or
* named datatype is specified for \p loc_id then the group will be
* created at the location where the attribute, dataset, or named
* datatype is attached. The name, name, must not already be taken by
* some other object and all parent groups must already exist.
*
* \p name can be a relative path based at \p loc_id or an absolute
* path from the root of the file. Use of this function requires that
* any intermediate groups specified in the path already exist.
*
* The length of a group name, or of the name of any object within a
* group, is not limited.
*
* \p size_hint is a hint for the number of bytes to reserve to store
* the names which will be eventually added to the new group. Passing a
* value of zero for \p size_hint is usually adequate since the library
* is able to dynamically resize the name heap, but a correct hint may
* result in better performance. If a non-positive value is supplied
* for \p size_hint, then a default size is chosen.
*
* The return value is a group identifier for the open group. This
* group identifier should be closed by calling H5Gclose() when it is
* no longer needed.
*
* See H5Gcreate_anon() for a discussion of the differences between
* H5Gcreate1() and H5Gcreate_anon().
*
* \par Example
* \snippet H5F_examples.c mount
*
* \version 1.8.0 Function H5Gcreate() renamed to H5Gcreate1() and deprecated
* in this release.
* \since 1.0.0
*
*/
H5_DLL hid_t H5Gcreate1(hid_t loc_id, const char *name, size_t size_hint);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Opens an existing group for modification and returns a group
* identifier for that group
*
* \fgdta_loc_id
* \param[in] name Name of the group to open
*
* \return \hid_t{group}
*
* \deprecated This function is deprecated in favor of H5Gopen2().
*
* \details H5Gopen1() opens an existing group, \p name, at the location
* specified by \p loc_id.
*
* H5Gopen1() returns a group identifier for the group that was
* opened. This group identifier should be released by calling
* H5Gclose() when it is no longer needed.
*
* \version 1.8.0 The function H5Gopen() was renamed to H5Gopen1()
* and deprecated in this release.
* \since 1.0.0
*
*/
H5_DLL hid_t H5Gopen1(hid_t loc_id, const char *name);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Creates a link of the specified type from \p new_name to \p
* cur_name
*
* \fg_loc_id{cur_loc_id}
* \param[in] type Link type
* \param[in] cur_name Name of the existing object
* \param[in] new_name New name for the object
*
* \return \herr_t
*
* \deprecated This function is deprecated.
*
* \details H5Glink() creates a new name for an object that has some current
* name, possibly one of many names it currently has.
*
* If \p link_type is #H5G_LINK_HARD, then \p cur_name must specify
* the name of an existing object and both names are interpreted
* relative to \p cur_loc_id, which is either a file identifier or a
* group identifier.
*
* If \p link_type is #H5G_LINK_SOFT, then \p cur_name can be anything
* and is interpreted at lookup time relative to the group which
* contains the final component of \p new_name. For instance, if \p
* cur_name is \Code{./foo}, \p new_name is \Code{./x/y/bar}, and a
* request is made for \Code{./x/y/bar}, then the actual object looked
* up is \Code{./x/y/./foo}.
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Glink(hid_t cur_loc_id, H5G_link_t type, const char *cur_name, const char *new_name);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Creates a link of the specified type from \p cur_name to \p
* new_name
*
* \fg_loc_id{cur_loc_id}
* \param[in] cur_name Name of the existing object
* \param[in] type Link type
* \fg_loc_id{new_loc_id}
* \param[in] new_name New name for the object
*
* \return \herr_t
*
* \deprecated This function is deprecated.
*
* \details H5Glink2() creates a new name for an object that has some current
* name, possibly one of many names it currently has.
*
* If \p link_type is #H5G_LINK_HARD, then \p cur_name must specify the
* name of an existing object and both names are interpreted relative
* to \p cur_loc_id and \p new_loc_id, respectively, which are either
* file identifiers or group identifiers.
*
* If \p link_type is #H5G_LINK_SOFT, then \p cur_name can be anything
* and is interpreted at lookup time relative to the group which
* contains the final component of \p new_name. For instance, if \p
* current_name is \Code{./foo}, \p new_name is \Code{./x/y/bar}, and a
* request is made for \Code{./x/y/bar}, then the actual object looked
* up is \Code{./x/y/./foo}.
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Glink2(hid_t cur_loc_id, const char *cur_name, H5G_link_t type, hid_t new_loc_id,
const char *new_name);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Renames an object within an HDF5 file
*
* \fg_loc_id{src_loc_id}
* \param[in] src_name Object's original name
* \param[in] dst_name Object's new name
*
* \return \herr_t
*
* \deprecated This function is deprecated.
*
* \details H5Gmove() renames an object within an HDF5 file. The original name,
* \p src_name, is unlinked from the group graph and the new name, \p
* dst_name, is inserted as an atomic operation. Both names are
* interpreted relative to \p loc_id, which is either a file or a group
* identifier.
*
* \attention Exercise care in moving groups as it is possible to render data in
* a file inaccessible with H5Gmove(). See The Group Interface in the
* HDF5 User's Guide.
*
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Gmove(hid_t src_loc_id, const char *src_name, const char *dst_name);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Renames an object within an HDF5 file
*
* \fg_loc_id{src_loc_id}
* \param[in] src_name Object's original name
* \fg_loc_id{dst_loc_id}
* \param[in] dst_name Object's new name
*
* \return \herr_t
*
* \deprecated This function is deprecated.
*
* \details H5Gmove2() renames an object within an HDF5 file. The original name,
* \p src_name, is unlinked from the group graph and the new name, \p
* dst_name, is inserted as an atomic operation.
*
* \p src_name and \p dst_name are interpreted relative to \p
* src_loc_id and \p dst_loc_id, respectively, which are either file or
* group identifiers.
*
* \attention Exercise care in moving groups as it is possible to render data in
* a file inaccessible with H5Gmove2(). See The Group Interface in the
* HDF5 User's Guide.
*
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Gmove2(hid_t src_loc_id, const char *src_name, hid_t dst_loc_id, const char *dst_name);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Removes the link to an object from a group
*
* \fg_loc_id{loc_id}
* \param[in] name Name of the object to unlink
*
* \return \herr_t
*
* \deprecated This function is deprecated in favor of the function H5Ldelete().
*
* \details H5Gunlink() removes the object specified by \p name from the group
* graph and decrements the link count for the object to which \p name
* points. This action eliminates any association between name and the
* object to which name pointed.
*
* Object headers keep track of how many hard links refer to an object;
* when the link count reaches zero, the object can be removed from the
* file. Objects which are open are not removed until all identifiers
* to the object are closed.
*
* If the link count reaches zero, all file space associated with the
* object will be released, i.e., identified in memory as freespace. If
* any object identifier is open for the object, the space will not be
* released until after the object identifier is closed.
*
* Note that space identified as freespace is available for re-use only
* as long as the file remains open; once a file has been closed, the
* HDF5 library loses track of freespace. See “Freespace Management” in
* the HDF5 User's Guide for further details.
*
* \attention Exercise care in moving groups as it is possible to render data in
* a file inaccessible with H5Gunlink(). See The Group Interface in the
* HDF5 User's Guide.
*
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Gunlink(hid_t loc_id, const char *name);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Returns the name of the object that the symbolic link points to
*
* \fg_loc_id{loc_id}
* \param[in] name Symbolic link to the object whose name is to be returned
* \param[in] size Maximum number of characters of value to be returned
* \param[out] buf A buffer to hold the name of the object being sought
*
* \return \herr_t
*
* \deprecated This function is deprecated in favor of the function H5Lget_val().
*
* \details H5Gget_linkval() returns up to size characters of the name of the
* object that the symbolic link name points to.
*
* The parameter \p loc_id is a file or group identifier.
*
* The parameter \p name must be a symbolic link pointing to the
* desired object and must be defined relative to \p loc_id.
*
* If size is smaller than the size of the returned object name, then
* the name stored in the buffer value will not be \c NULL terminated.
*
* This function fails if \p name is not a symbolic link. The presence
* of a symbolic link can be tested by passing zero for \p size and \p
* NULL for value.
*
* This function should be used only after H5Lget_info1() (or the
* deprecated function H5Gget_objinfo()) has been called to verify that
* name is a symbolic link.
*
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Gget_linkval(hid_t loc_id, const char *name, size_t size, char *buf /*out*/);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Sets comment for specified object
*
* \fgdt_loc_id
* \param[in] name Name of the object whose comment is to be set or reset
* name must be \Code{'.'} (dot) if \p loc_id fully specifies
* the object for which the comment is to be set.
* \param[in] comment The new comment
*
* \return \herr_t
*
* \deprecated This function is deprecated in favor of the function
* H5Oset_comment().
*
* \details H5Gset_comment() sets the comment for the object specified by \p
* loc_id and name to comment. Any previously existing comment is
* overwritten.
*
* \p loc_id can specify any object in the file. name can be one of the
* following:
* \li The name of the object relative to \p loc_id
* \li An absolute name of the object, starting from \c /, the file’s
* root group
* \li A dot (\c .), if \p loc_id fully specifies the object
*
* If \p comment is the empty string or a null pointer, the comment
* message is removed from the object.
*
* Comments should be relatively short, null-terminated, ASCII strings.
*
* Comments can be attached to any object that has an object header,
* e.g., datasets, groups, and named datatypes, but not symbolic links.
*
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Gset_comment(hid_t loc_id, const char *name, const char *comment);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Retrieves comment for specified object
*
* \fgdt_loc_id
* \param[in] name Name of the object whose comment is to be set or reset
* name must be \Code{'.'} (dot) if \p loc_id fully specifies
* the object for which the comment is to be set.
* \param[in] bufsize Maximum number of comment characters to be returned in \p buf.
* \param[in] buf The comment
*
* \return Returns the number of characters in the comment, counting the \c NULL
* terminator, if successful; the value returned may be larger than
* \p bufsize. Otherwise returns a negative value.
*
* \deprecated This function is deprecated in favor of the function
* H5Oget_comment().
*
* \details H5Gget_comment() retrieves the comment for the the object specified
* by \p loc_id and \p name. The comment is returned in the buffer \p
* buf.
*
* \p loc_id can specify any object in the file. name can be one of the
* following:
* \li The name of the object relative to \p loc_id
* \li An absolute name of the object, starting from \c /, the file’s
* root group
* \li A dot (\c .), if \p loc_id fully specifies the object
*
* At most bufsize characters, including a null-terminator, are
* returned in \p buf. The returned value is not null-terminated if the
* comment is longer than the supplied buffer. If the size of the
* comment is unknown, a preliminary \p H5Gget_comment() call will
* return the size of the comment, including space for the
* null-terminator.
*
* If an object does not have a comment, the empty string is returned
* in comment.
*
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL int H5Gget_comment(hid_t loc_id, const char *name, size_t bufsize, char *buf);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Iterates over the entries of a group invoking a callback for each
* entry encountered
*
* \fg_loc_id
* \param[in] name Group over which the iteration is performed
* \param[in,out] idx Location at which to begin the iteration
* \param[in] op Operation to be performed on an object at each step of the
* iteration
* \param[in,out] op_data Data associated with the operation
*
* \return \herr_t
*
* \deprecated This function is deprecated in favor of the function
* H5Literate1().
*
* \details H5Giterate() iterates over the members of name in the file or group
* specified with \p loc_id. For each object in the group, the \p
* op_data and some additional information, specified below, are passed
* to the operator function. The iteration begins with the \p idx
* object in the group and the next element to be processed by the
* operator is returned in \p idx. If \p idx is NULL, then the iterator
* starts at the first group member; since no stopping point is
* returned in this case, the iterator cannot be restarted if one of
* the calls to its operator returns non-zero. H5Giterate() does not
* recursively follow links into subgroups of the specified group.
*
* The prototype for \ref H5G_iterate_t is:
* \snippet this H5G_iterate_t_snip
*
* The operation receives the group identifier for the group being
* iterated over, \p group, the name of the current object within
* the group, \p name, and the pointer to the operator data
* passed in to H5Giterate(), \p op_data.
*
* The return values from an operator are:
* \li Zero causes the iterator to continue, returning zero when all
* group members have been processed.
* \li Positive causes the iterator to immediately return that positive
* value, indicating short-circuit success. The iterator can be
* restarted at the next group member.
* \li Negative causes the iterator to immediately return that value,
* indicating failure. The iterator can be restarted at the next
* group member.
*
* H5Giterate() assumes that the membership of the group identified by
* \p name remains unchanged through the iteration. If the membership
* changes during the iteration, the function's behavior is undefined.
*
* H5Giterate() is not recursive. In particular, if a member of \p name
* is found to be a group, call it \c subgroup_a, H5Giterate() does not
* examine the members of \c subgroup_a. When recursive iteration is
* required, the application must handle the recursion, explicitly
* calling H5Giterate() on discovered subgroups.
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Giterate(hid_t loc_id, const char *name, int *idx, H5G_iterate_t op, void *op_data);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Returns number of objects in the group specified by its identifier
*
* \fg_loc_id
* \param[out] num_objs Number of objects in the group
*
* \return \herr_t
*
* \deprecated This function is deprecated in favor of the function H5Gget_info().
*
* \details H5Gget_num_objs() returns number of objects in a group. Group is
* specified by its identifier \p loc_id. If a file identifier is
* passed in, then the number of objects in the root group is returned.
*
* \version 1.8.0 Function deprecated in this release.
*
*/
H5_DLL herr_t H5Gget_num_objs(hid_t loc_id, hsize_t *num_objs);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Returns information about an object.
*
* \fgdt_loc_id
* \param[in] name Name of the object for which status is being sought
* \param[in] follow_link Link flag
* \param[out] statbuf Buffer in which to return information about the object
*
* \return \herr_t
*
* \deprecated This function is deprecated in favor of the functions H5Oget_info()
* and H5Lget_info1().
*
* \details H5Gget_objinfo() returns information about the specified object
* through the \p statbuf argument.
*
* A file or group identifier, \p loc_id, and an object name, \p name,
* relative to \p loc_id, are commonly used to specify the
* object. However, if the object identifier is already known to the
* application, an alternative approach is to use that identifier, \c
* obj_id, in place of \p loc_id, and a dot (\c .) in place of \p
* name. Thus, the alternative versions of the first portion of an
* H5Gget_objinfo() call would be as follows:
* \code
* H5Gget_objinfo (loc_id name ...)
* H5Gget_objinfo (obj_id . ...)
* \endcode
*
* If the object is a symbolic link and follow_link is zero (0), then
* the information returned describes the link itself; otherwise the
* link is followed and the information returned describes the object
* to which the link points. If \p follow_link is non-zero but the
* final symbolic link is dangling (does not point to anything), then
* an error is returned. The \p statbuf fields are undefined for an
* error. The existence of an object can be tested by calling this
* function with a \c NULL \p statbuf.
*
* H5Gget_objinfo() fills in the following data structure (defined in
* H5Gpublic.h):
* \snippet this H5G_stat_t_snip
*
* where \ref H5O_stat_t (defined in H5Opublic.h) is:
* \snippet H5Opublic.h H5O_stat_t_snip
*
* \attention Some systems will be able to record the time accurately but unable
* to retrieve the correct time; such systems (e.g., Irix64) will
* report an \c mtime value of 0 (zero).
*
* \version 1.8.0 Function deprecated in this release.
* \version 1.6.1 Two new fields were added to the \ref H5G_stat_t struct in
* this release.
*
*/
H5_DLL herr_t H5Gget_objinfo(hid_t loc_id, const char *name, hbool_t follow_link,
H5G_stat_t *statbuf /*out*/);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Returns a name of an object specified by an index
*
* \fg_loc_id
* \param[in] idx Transient index identifying object
* \param[in,out] name Pointer to user-provided buffer the object name
* \param[in] size Name length
*
* \return Returns the size of the object name if successful, or 0 if no name is
* associated with the group identifier. Otherwise returns a negative
* value.
*
* \deprecated This function is deprecated in favor of the function H5Lget_name_by_idx().
*
* \details H5Gget_objname_by_idx() returns a name of the object specified by
* the index \p idx in the group \p loc_id.
*
* The group is specified by a group identifier \p loc_id. If
* preferred, a file identifier may be passed in \p loc_id; that file's
* root group will be assumed.
*
* \p idx is the transient index used to iterate through the objects in
* the group. The value of \p idx is any nonnegative number less than
* the total number of objects in the group, which is returned by the
* function H5Gget_num_objs(). Note that this is a transient index; an
* object may have a different index each time a group is opened.
*
* The object name is returned in the user-specified buffer \p name.
*
* If the size of the provided buffer \p name is less or equal the
* actual object name length, the object name is truncated to
* \Code{max_size - 1} characters.
*
* Note that if the size of the object's name is unkown, a preliminary
* call to H5Gget_objname_by_idx() with \p name set to \c NULL will
* return the length of the object's name. A second call to
* H5Gget_objname_by_idx() can then be used to retrieve the actual
* name.
*
* \version 1.8.0 Function deprecated in this release.
* \since 1.6.0
*
*/
H5_DLL ssize_t H5Gget_objname_by_idx(hid_t loc_id, hsize_t idx, char *name, size_t size);
/**
*-------------------------------------------------------------------------
* \ingroup H5G
*
* \brief Returns the type of an object specified by an index
*
* \fg_loc_id
* \param[in] idx Transient index identifying object
*
* \return Returns the type of the object if successful. Otherwise returns a
* negative value.
*
* \deprecated This function is deprecated in favor of the function H5Oget_info().
*
* \details H5Gget_objtype_by_idx() returns the type of the object specified by
* the index \p idx in the group \p loc_id.
*
* The group is specified by a group identifier \p loc_id. If
* preferred, a file identifier may be passed in \p loc_id; that file's
* root group will be assumed.
*
* \p idx is the transient index used to iterate through the objects in
* the group. This parameter is described in more detail in the
* discussion of H5Gget_objname_by_idx().
*
* \version 1.8.0 Function deprecated in this release.
* \version 1.6.0 The function return type changed from \c int to the enumerated
* type \ref H5G_obj_t.
* \since 1.6.0
*
*/
H5_DLL H5G_obj_t H5Gget_objtype_by_idx(hid_t loc_id, hsize_t idx);
#endif /* H5_NO_DEPRECATED_SYMBOLS */
src/H5Ipublic.h
+12 -10
View File
@@ -32,6 +32,7 @@
* test/tmisc.c to verify that the H5I{inc|dec|get}_ref() routines
* work correctly with it. \endinternal
*/
//! <!-- [H5I_type_t_snip] -->
typedef enum H5I_type_t {
H5I_UNINIT = (-2), /**< uninitialized type */
H5I_BADID = (-1), /**< invalid Type */
@@ -53,6 +54,7 @@ typedef enum H5I_type_t {
H5I_EVENTSET, /**< type ID for event sets */
H5I_NTYPES /**< number of library types, MUST BE LAST! */
} H5I_type_t;
//! <!-- [H5I_type_t_snip] -->
/**
* Type of IDs to return to users
@@ -86,30 +88,30 @@ typedef herr_t (*H5I_free_t)(void *, void **);
/**
* The type of a function to compare objects & keys
*/
//! [H5I_search_func_t_snip]
//! <!-- [H5I_search_func_t_snip] -->
typedef int (*H5I_search_func_t)(void *obj, hid_t id, void *key);
//! [H5I_search_func_t_snip]
//! <!-- [H5I_search_func_t_snip] -->
/**
* The type of H5Iiterate() callback functions
*/
//! [H5I_iterate_func_t_snip]
//! <!-- [H5I_iterate_func_t_snip] -->
typedef herr_t (*H5I_iterate_func_t)(hid_t id, void *udata);
//! [H5I_iterate_func_t_snip]
//! <!-- [H5I_iterate_func_t_snip] -->
/**
* The type of the realize_cb callback for H5Iregister_future
*/
//! [H5I_future_realize_func_t_snip]
//! <!-- [H5I_future_realize_func_t_snip] -->
typedef herr_t (*H5I_future_realize_func_t)(void *future_object, hid_t *actual_object_id);
//! [H5I_future_realize_func_t_snip]
//! <!-- [H5I_future_realize_func_t_snip] -->
/**
* The type of the discard_cb callback for H5Iregister_future
*/
//! [H5I_future_discard_func_t_snip]
//! <!-- [H5I_future_discard_func_t_snip] -->
typedef herr_t (*H5I_future_discard_func_t)(void *future_object);
//! [H5I_future_discard_func_t_snip]
//! <!-- [H5I_future_discard_func_t_snip] -->
#ifdef __cplusplus
extern "C" {
@@ -173,7 +175,7 @@ H5_DLL hid_t H5Iregister(H5I_type_t type, const void *object);
*
* \details The \p realize_cb parameter is a function pointer that will be
* invoked by the HDF5 library to convert a future object into an
* actual object. The \realize_cb function may be invoked by
* actual object. The \p realize_cb function may be invoked by
* H5Iobject_verify() to return the actual object for a user-defined
* ID class (i.e. an ID class registered with H5Iregister_type()) or
* internally by the HDF5 library in order to use or get information
@@ -281,7 +283,7 @@ H5_DLL void *H5Iremove_verify(hid_t id, H5I_type_t type);
* \p id.
*
* Valid types returned by the function are:
* \types
* \id_types
*
* If no valid type can be determined or the identifier submitted is
* invalid, the function returns #H5I_BADID.
src/H5Lmodule.h
+2
View File
@@ -35,6 +35,8 @@
*
* \defgroup TRAV Link Traversal
* \ingroup H5L
* \defgroup H5LA Advanced Link Functions
* \ingroup H5L
*/
#endif /* H5Lmodule_H */
src/H5Lpublic.h
+43 -33
View File
@@ -92,7 +92,7 @@ typedef enum {
/**
* \brief Information struct for links
*/
//! [H5L_info2_t_snip]
//! <!-- [H5L_info2_t_snip] -->
typedef struct {
H5L_type_t type; /**< Type of link */
hbool_t corder_valid; /**< Indicate if creation order is valid */
@@ -103,7 +103,7 @@ typedef struct {
size_t val_size; /**< Size of a soft link or user-defined link value */
} u;
} H5L_info2_t;
//! [H5L_info2_t_snip]
//! <!-- [H5L_info2_t_snip] -->
/* The H5L_class_t struct can be used to override the behavior of a
* "user-defined" link class. Users should populate the struct with callback
@@ -150,7 +150,7 @@ typedef ssize_t (*H5L_query_func_t)(const char *link_name, const void *lnkdata,
* "user-defined" link class. Users should populate the struct with callback
* functions defined elsewhere.
*/
//! [H5L_class_t_snip]
//! <!-- [H5L_class_t_snip] -->
typedef struct {
int version; /**< Version number of this struct */
H5L_type_t id; /**< Link type ID */
@@ -162,16 +162,16 @@ typedef struct {
H5L_delete_func_t del_func; /**< Callback for link deletion */
H5L_query_func_t query_func; /**< Callback for queries */
} H5L_class_t;
//! [H5L_class_t_snip]
//! <!-- [H5L_class_t_snip] -->
/**
* \brief Prototype for H5Literate2(), H5Literate_by_name2() operator
*
* The H5O_token_t version is used in the VOL layer and future public API calls.
*/
//! [H5L_iterate2_t_snip]
//! <!-- [H5L_iterate2_t_snip] -->
typedef herr_t (*H5L_iterate2_t)(hid_t group, const char *name, const H5L_info2_t *info, void *op_data);
//! [H5L_iterate2_t_snip]
//! <!-- [H5L_iterate2_t_snip] -->
/**
* \brief Callback for external link traversal
@@ -201,8 +201,6 @@ typedef herr_t (*H5L_elink_traverse_t)(const char *parent_file_name, const char
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Lmove() moves a link within an HDF5 file. The original link,
* \p src_name, is removed from \p src_loc and the new link,
* \p dst_name, is inserted at dst_loc. This change is
@@ -321,8 +319,6 @@ H5_DLL herr_t H5Lcopy(hid_t src_loc, const char *src_name, hid_t dst_loc, const
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Lcreate_hard() creates a new hard link to a pre-existing object
* in an HDF5 file.
*
@@ -357,6 +353,11 @@ H5_DLL herr_t H5Lcopy(hid_t src_loc, const char *src_name, hid_t dst_loc, const
*/
H5_DLL herr_t H5Lcreate_hard(hid_t cur_loc, const char *cur_name, hid_t dst_loc, const char *dst_name,
hid_t lcpl_id, hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Lcreate_hard}
*/
H5_DLL herr_t H5Lcreate_hard_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t cur_loc_id, const char *cur_name, hid_t new_loc_id,
const char *new_name, hid_t lcpl_id, hid_t lapl_id, hid_t es_id);
@@ -373,8 +374,6 @@ H5_DLL herr_t H5Lcreate_hard_async(const char *app_file, const char *app_func, u
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Lcreate_soft() creates a new soft link to an object in an HDF5
* file.
*
@@ -426,6 +425,11 @@ H5_DLL herr_t H5Lcreate_hard_async(const char *app_file, const char *app_func, u
*/
H5_DLL herr_t H5Lcreate_soft(const char *link_target, hid_t link_loc_id, const char *link_name, hid_t lcpl_id,
hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Lcreate_soft}
*/
H5_DLL herr_t H5Lcreate_soft_async(const char *app_file, const char *app_func, unsigned app_line,
const char *link_target, hid_t link_loc_id, const char *link_name,
hid_t lcpl_id, hid_t lapl_id, hid_t es_id);
@@ -440,8 +444,6 @@ H5_DLL herr_t H5Lcreate_soft_async(const char *app_file, const char *app_func, u
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Ldelete() removes the link specified by \p name from the location
* \p loc_id.
*
@@ -468,6 +470,11 @@ H5_DLL herr_t H5Lcreate_soft_async(const char *app_file, const char *app_func, u
*
*/
H5_DLL herr_t H5Ldelete(hid_t loc_id, const char *name, hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Ldelete}
*/
H5_DLL herr_t H5Ldelete_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t lapl_id, hid_t es_id);
/**
@@ -484,8 +491,6 @@ H5_DLL herr_t H5Ldelete_async(const char *app_file, const char *app_func, unsign
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Ldelete_by_idx() removes the \Emph{n}-th link in a group
* according to the specified order, \p order, in the specified index,
* \p index.
@@ -500,6 +505,11 @@ H5_DLL herr_t H5Ldelete_async(const char *app_file, const char *app_func, unsign
*/
H5_DLL herr_t H5Ldelete_by_idx(hid_t loc_id, const char *group_name, H5_index_t idx_type,
H5_iter_order_t order, hsize_t n, hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Ldelete_by_idx}
*/
H5_DLL herr_t H5Ldelete_by_idx_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *group_name, H5_index_t idx_type,
H5_iter_order_t order, hsize_t n, hid_t lapl_id, hid_t es_id);
@@ -516,8 +526,6 @@ H5_DLL herr_t H5Ldelete_by_idx_async(const char *app_file, const char *app_func,
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Lget_val() returns tha value of link \p name. For smbolic links,
* this is the path to which the link points, including the null
* terminator. For external and user-defined links, it is the link
@@ -575,8 +583,6 @@ H5_DLL herr_t H5Lget_val(hid_t loc_id, const char *name, void *buf /*out*/, size
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Lget_val_by_idx() retrieves the value of the \Emph{n}-th link in
* a group, according to the specified order, \p order, within an
* index, \p index.
@@ -630,8 +636,6 @@ H5_DLL herr_t H5Lget_val_by_idx(hid_t loc_id, const char *group_name, H5_index_t
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Lexists() allows an application to determine whether the link \p
* name exists in the location specified by \p loc_id. The link may be
* of any type; only the presence of a link with that name is checked.
@@ -707,6 +711,11 @@ H5_DLL herr_t H5Lget_val_by_idx(hid_t loc_id, const char *group_name, H5_index_t
*
*/
H5_DLL htri_t H5Lexists(hid_t loc_id, const char *name, hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Lexists}
*/
H5_DLL herr_t H5Lexists_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hbool_t *exists, hid_t lapl_id, hid_t es_id);
/**
@@ -721,8 +730,6 @@ H5_DLL herr_t H5Lexists_async(const char *app_file, const char *app_func, unsign
*
* \return \herr_t
*
* \todo We need to get the location ID story straight!
*
* \details H5Lget_info2() returns information about the specified link through
* the \p linfo argument.
*
@@ -830,8 +837,6 @@ H5_DLL herr_t H5Lget_info2(hid_t loc_id, const char *name, H5L_info2_t *linfo, h
*
* \see H5Lget_info2()
*
* \todo Document H5Lget_info_by_idx()
*
*/
H5_DLL herr_t H5Lget_info_by_idx2(hid_t loc_id, const char *group_name, H5_index_t idx_type,
H5_iter_order_t order, hsize_t n, H5L_info2_t *linfo, hid_t lapl_id);
@@ -957,6 +962,11 @@ H5_DLL ssize_t H5Lget_name_by_idx(hid_t loc_id, const char *group_name, H5_index
*/
H5_DLL herr_t H5Literate2(hid_t grp_id, H5_index_t idx_type, H5_iter_order_t order, hsize_t *idx,
H5L_iterate2_t op, void *op_data);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Literate}
*/
H5_DLL herr_t H5Literate_async(const char *app_file, const char *app_func, unsigned app_line, hid_t group_id,
H5_index_t idx_type, H5_iter_order_t order, hsize_t *idx_p, H5L_iterate2_t op,
void *op_data, hid_t es_id);
@@ -1642,8 +1652,10 @@ H5_DLL herr_t H5Lcreate_external(const char *file_name, const char *obj_name, hi
/* Typedefs */
/* Information struct for link (for H5Lget_info1/H5Lget_info_by_idx1) */
//! [H5L_info1_t_snip]
//! <!-- [H5L_info1_t_snip] -->
/**
* Information struct for link (for H5Lget_info1() and H5Lget_info_by_idx1())
*/
typedef struct {
H5L_type_t type; /**< Type of link */
hbool_t corder_valid; /**< Indicate if creation order is valid */
@@ -1654,7 +1666,7 @@ typedef struct {
size_t val_size; /**< Size of a soft link or UD link value */
} u;
} H5L_info1_t;
//! [H5L_info1_t_snip]
//! <!-- [H5L_info1_t_snip] -->
/** Callback during link traversal */
typedef hid_t (*H5L_traverse_0_func_t)(const char *link_name, hid_t cur_group, const void *lnkdata,
@@ -1674,9 +1686,9 @@ typedef struct {
} H5L_class_0_t;
/** Prototype for H5Literate1() / H5Literate_by_name1() operator */
//! [H5L_iterate1_t_snip]
//! <!-- [H5L_iterate1_t_snip] -->
typedef herr_t (*H5L_iterate1_t)(hid_t group, const char *name, const H5L_info1_t *info, void *op_data);
//! [H5L_iterate1_t_snip]
//! <!-- [H5L_iterate1_t_snip] -->
/* Function prototypes */
/**
@@ -1694,8 +1706,6 @@ typedef herr_t (*H5L_iterate1_t)(hid_t group, const char *name, const H5L_info1_
* \deprecated As of HDF5-1.12 this function has been deprecated in favor of
* the function H5Lget_info2() or the macro H5Lget_info().
*
* \todo We need to get the location ID story straight!
*
* \details H5Lget_info1() returns information about the specified link through
* the \p linfo argument.
*
src/H5MMpublic.h
+5
View File
@@ -29,8 +29,13 @@
#include "H5public.h"
/* These typedefs are currently used for VL datatype allocation/freeing */
//! <!-- [H5MM_allocate_t_snip] -->
typedef void *(*H5MM_allocate_t)(size_t size, void *alloc_info);
//! <!-- [H5MM_allocate_t_snip] -->
//! <!-- [H5MM_free_t_snip] -->
typedef void (*H5MM_free_t)(void *mem, void *free_info);
//! <!-- [H5MM_free_t_snip] -->
#ifdef __cplusplus
extern "C" {
src/H5Mmodule.h
+45
View File
@@ -26,4 +26,49 @@
#define H5_MY_PKG_ERR H5E_MAP
#define H5_MY_PKG_INIT YES
/**
* \defgroup H5M H5M
* \brief Map Interface
*
* \details \Bold{The interface can only be used with the HDF5 VOL connectors that
* implement map objects.} The native HDF5 library does not support this
* feature.
*
* While the HDF5 data model is a flexible way to store data, some
* applications require a more general way to index information. HDF5
* effectively uses key-value stores internally for a variety of
* purposes, but it does not expose a generic key-value store to the
* API. The Map APIs provide this capability to the HDF5 applications
* in the form of HDF5 map objects. These Map objects contain
* application-defined key-value stores, to which key-value pairs can
* be added, and from which values can be retrieved by key.
*
* HDF5 VOL connectors with support for map objects:
* - DAOS
*
* \par Example:
* \code
* hid_t file_id, fapl_id, map_id, vls_type_id;
* const char *names[2] = ["Alice", "Bob"];
* uint64_t IDs[2] = [25385486, 34873275];
* uint64_t val_out;
*
* <HDF5 VOL setup code ....>
*
* vls_type_id = H5Tcopy(H5T_C_S1);
* H5Tset_size(vls_type_id, H5T_VARIABLE);
* file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);
* map_id = H5Mcreate(file_id, "map", vls_type_id, H5T_NATIVE_UINT64, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
* H5Mput(map_id, vls_type_id, &names[0], H5T_NATIVE_UINT64, &IDs[0], H5P_DEFAULT);
* H5Mput(map_id, vls_type_id, &names[1], H5T_NATIVE_UINT64, &IDs[1], H5P_DEFAULT);
* H5Mget(map_id, vls_type_id, &names[0], H5T_NATIVE_UINT64, &val_out, H5P_DEFAULT);
* if(val_out != IDs[0])
* ERROR;
* H5Mclose(map_id);
* H5Tclose(vls_type_id);
* H5Fclose(file_id);
* \endcode
*
*/
#endif /* H5Dmodule_H */
src/H5Mpublic.h
+368 -5
View File
@@ -61,8 +61,12 @@ typedef enum H5VL_map_specific_t {
H5VL_MAP_DELETE /* H5Mdelete */
} H5VL_map_specific_t;
/* Callback for H5Miterate() */
//! <!-- [H5M_iterate_t_snip] -->
/**
* Callback for H5Miterate()
*/
typedef herr_t (*H5M_iterate_t)(hid_t map_id, const void *key, void *op_data);
//! <!-- [H5M_iterate_t_snip] -->
/********************/
/* Public Variables */
@@ -81,38 +85,397 @@ extern "C" {
*/
#ifdef H5_HAVE_MAP_API
/**
* \ingroup H5M
*
* \brief Creates a map object
*
* \fgdta_loc_id
* \param[in] name Map object name
* \type_id{key_type_id}
* \type_id{val_type_id}
* \lcpl_id
* \mcpl_id
* \mapl_id
* \returns \hid_t{map object}
*
* \details H5Mcreate() creates a new map object for storing key-value
* pairs. The in-file datatype for keys is defined by \p key_type_id
* and the in-file datatype for values is defined by \p val_type_id. \p
* loc_id specifies the location to create the the map object and \p
* name specifies the name of the link to the map object relative to
* \p loc_id.
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5Mcreate(hid_t loc_id, const char *name, hid_t key_type_id, hid_t val_type_id, hid_t lcpl_id,
hid_t mcpl_id, hid_t mapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Mcreate}
*/
H5_DLL hid_t H5Mcreate_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t key_type_id, hid_t val_type_id, hid_t lcpl_id,
hid_t mcpl_id, hid_t mapl_id, hid_t es_id);
/**
* \ingroup H5M
*
* \brief
*
* \details
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5Mcreate_anon(hid_t loc_id, hid_t key_type_id, hid_t val_type_id, hid_t mcpl_id, hid_t mapl_id);
/**
* \ingroup H5M
*
* \brief Opens a map object
*
* \fgdta_loc_id{loc_id}
* \param[in] name Map object name relative to \p loc_id
* \mapl_id
* \returns \hid_t{map object}
*
* \details H5Mopen() finds a map object specified by \p name under the location
* specified by \p loc_id. The map object should be close with
* H5Mclose() when the application is not longer interested in
* accessing it.
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5Mopen(hid_t loc_id, const char *name, hid_t mapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Mopen}
*/
H5_DLL hid_t H5Mopen_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t mapl_id, hid_t es_id);
/**
* \ingroup H5M
*
* \brief Terminates access to a map object
*
* \map_id
* \returns \herr_t
*
* \details H5Mclose() closes access to a map object specified by \p map_id and
* releases resources used by it.
*
* It is illegal to subsequently use that same map identifier in calls
* to other map functions.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Mclose(hid_t map_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Mclose}
*/
H5_DLL herr_t H5Mclose_async(const char *app_file, const char *app_func, unsigned app_line, hid_t map_id,
hid_t es_id);
H5_DLL hid_t H5Mget_key_type(hid_t map_id);
H5_DLL hid_t H5Mget_val_type(hid_t map_id);
H5_DLL hid_t H5Mget_create_plist(hid_t map_id);
H5_DLL hid_t H5Mget_access_plist(hid_t map_id);
/**
* \ingroup H5M
*
* \brief Gets key datatype for a map object
*
* \map_id
* \returns \hid_t{datatype}
*
* \details H5Mget_key_type() retrieves key datatype as stored in the file for a
* map object specified by \p map_id and returns identifier for the
* datatype.
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5Mget_key_type(hid_t map_id);
/**
* \ingroup H5M
*
* \brief Gets value datatype for a map object
*
* \map_id
* \returns \hid_t{datatype}
*
* \details H5Mget_val_type() retrieves value datatype as stored in the file for
* a map object specified by \p map_id and returns identifier for the
* datatype .
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5Mget_val_type(hid_t map_id);
/**
* \ingroup H5M
*
* \brief Gets creation property list for a map object
*
* \map_id
* \returns \hid_t{map creation property list}
*
* \details H5Mget_create_plist() returns an identifier for a copy of the
* creation property list for a map object specified by \p map_id.
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5Mget_create_plist(hid_t map_id);
/**
* \ingroup H5M
*
* \brief Gets access property list for a map object
*
* \map_id
* \returns \hid_t{map access property list}
*
* \details H5Mget_access_plist() returns an identifier for a copy of the access
* property list for a map object specified by \p map_id.
*
* \since 1.13.0
*
*/
H5_DLL hid_t H5Mget_access_plist(hid_t map_id);
/**
* \ingroup H5M
*
* \brief Retrieves the number of key-value pairs in a map object
*
* \map_id
* \param[out] count The number of key-value pairs stored in the map object
* \dxpl_id
* \returns \herr_t
*
* \details H5Mget_count() retrieves the number of key-value pairs stored in a
* map specified by map_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Mget_count(hid_t map_id, hsize_t *count, hid_t dxpl_id);
/**
* \ingroup H5M
*
* \brief Adds a key-value pair to a map object
*
* \map_id
* \type_id{key_mem_type_id}
* \param[in] key Pointer to key buffer
* \type_id{val_mem_type_id}
* \param[in] value Pointer to value buffer
* \dxpl_id
* \returns \herr_t
*
* \details H5Mput() adds a key-value pair to a map object specified by \p
* map_id, or updates the value for the specified key if one was set
* previously.
*
* \p key_mem_type_id and \p val_mem_type_id specify the datatypes for
* the provided key and value buffers, and if different from those used
* to create the map object, the key and value will be internally
* converted to the datatypes for the map object.
*
* Any further options can be specified through the property list
* \p dxpl_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Mput(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id,
const void *value, hid_t dxpl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Mput}
*/
H5_DLL herr_t H5Mput_async(const char *app_file, const char *app_func, unsigned app_line, hid_t map_id,
hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id, const void *value,
hid_t dxpl_id, hid_t es_id);
/**
* \ingroup H5M
*
* \brief Retrieves a key-value pair from a map object
*
* \map_id
* \type_id{key_mem_type_id}
* \param[in] key Pointer to key buffer
* \type_id{val_mem_type_id}
* \param[out] value Pointer to value buffer
* \dxpl_id
* \returns \herr_t
*
* \details H5Mget() retrieves from a map object specified by \p map_id, the
* value associated with the provided key \p key. \p key_mem_type_id
* and \p val_mem_type_id specify the datatypes for the provided key
* and value buffers. If if the datatype specified by \p
* key_mem_type_id is different from that used to create the map object
* the key will be internally converted to the datatype for the map
* object for the query, and if the datatype specified by \p
* val_mem_type_id is different from that used to create the map object
* the returned value will be converted to have a datatype as specified
* by \p val_mem_type_id before the function returns.
*
* Any further options can be specified through the property list
* \p dxpl_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Mget(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id, void *value,
hid_t dxpl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Mget}
*/
H5_DLL herr_t H5Mget_async(const char *app_file, const char *app_func, unsigned app_line, hid_t map_id,
hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id, void *value,
hid_t dxpl_id, hid_t es_id);
/**
* \ingroup H5M
*
* \brief Checks if provided key exists in a map object
*
* \map_id
* \type_id{key_mem_type_id}
* \param[in] key Pointer to key buffer
* \param[out] exists Pointer to a buffer to return the existence status
* \dxpl_id
* \returns \herr_t
*
* \details H5Mexists() checks if the provided key is stored in the map object
* specified by \p map_id. If \p key_mem_type_id is different from that
* used to create the map object the key will be internally converted
* to the datatype for the map object for the query.
*
* Any further options can be specified through the property list
* \p dxpl_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Mexists(hid_t map_id, hid_t key_mem_type_id, const void *key, hbool_t *exists, hid_t dxpl_id);
/**
* \ingroup H5M
*
* \brief Iterates over all key-value pairs in a map object
*
* \map_id
* \param[in,out] idx iteration index
* \type_id{key_mem_type_id}
* \param[in] op User-defined iterator function
* \op_data
* \dxpl_id
* \returns \herr_t
*
* \details H5Miterate() iterates over all key-value pairs stored in the map
* object specified by \p map_id, making the callback specified by \p
* op for each. The \p idx parameter is an in/out parameter that may be
* used to restart a previously interrupted iteration. At the start of
* iteration \p idx should be set to 0, and to restart iteration at the
* same location on a subsequent call to H5Miterate(), \p idx should be
* the same value as returned by the previous call. Iterate callback is
* defined as:
* \snippet this H5M_iterate_t_snip
* The \p key parameter is the buffer for the key for this iteration,
* converted to the datatype specified by \p key_mem_type_id. The \p
* op_data parameter is a simple pass through of the value passed to
* H5Miterate(), which can be used to store application-defined data for
* iteration. A negative return value from this function will cause
* H5Miterate() to issue an error, while a positive return value will
* cause H5Miterate() to stop iterating and return this value without
* issuing an error. A return value of zero allows iteration to continue.
*
* Any further options can be specified through the property list \p dxpl_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Miterate(hid_t map_id, hsize_t *idx, hid_t key_mem_type_id, H5M_iterate_t op, void *op_data,
hid_t dxpl_id);
/**
* \ingroup H5M
*
* \brief Iterates over all key-value pairs in a map object
*
* \loc_id
* \param[in] map_name Map object name relative to the location specified by \p loc_id
* \param[in,out] idx Iteration index
* \type_id{key_mem_type_id}
* \param[in] op User-defined iterator function
* \op_data
* \dxpl_id
* \lapl_id
* \returns \herr_t
*
* \details H5Miterate_by_name() iterates over all key-value pairs stored in the
* map object specified by \p map_id, making the callback specified by
* \p op for each. The \p idx parameter is an in/out parameter that may
* be used to restart a previously interrupted iteration. At the start
* of iteration \p idx should be set to 0, and to restart iteration at
* the same location on a subsequent call to H5Miterate(), \p idx
* should be the same value as returned by the previous call. Iterate
* callback is defined as:
* \snippet this H5M_iterate_t_snip
* The\p key parameter is the buffer for the key for this iteration,
* converted to the datatype specified by \p key_mem_type_id. The \p
* op_data parameter is a simple pass through of the value passed to
* H5Miterate(), which can be used to store application-defined data
* for iteration. A negative return value from this function will cause
* H5Miterate() to issue an error, while a positive return value will cause
* H5Miterate() to stop iterating and return this value without issuing an
* error. A return value of zero allows iteration to continue.
*
* Any further options can be specified through the property list \p dxpl_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Miterate_by_name(hid_t loc_id, const char *map_name, hsize_t *idx, hid_t key_mem_type_id,
H5M_iterate_t op, void *op_data, hid_t dxpl_id, hid_t lapl_id);
/**
* \ingroup H5M
*
* \brief Deletes a key-value pair from a map object
*
* \map_id
* \type_id{key_mem_type_id}
* \param[in] key Pointer to key buffer
* \dxpl_id
* \returns \herr_t
*
* \details H5Mdelete() deletes a key-value pair from the map object specified
* by \p map_id. \p key_mem_type_id specifies the datatype for the
* provided key buffer key, and if different from that used to create
* the map object, the key will be internally converted to the datatype
* for the map object.
*
* Any further options can be specified through the property list \p dxpl_id.
*
* \since 1.13.0
*
*/
H5_DLL herr_t H5Mdelete(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t dxpl_id);
/* API Wrappers for async routines */
src/H5Opublic.h
+136 -91
View File
@@ -86,17 +86,15 @@
#define H5O_INFO_NUM_ATTRS 0x0004u /* Fill in the num_attrs field */
#define H5O_INFO_ALL (H5O_INFO_BASIC | H5O_INFO_TIME | H5O_INFO_NUM_ATTRS)
/* Flags for H5Oget_native_info.
* Theses flags determine which fields will be filled in in the H5O_native_info_t
* struct.
//! <!-- [H5O_native_info_fields_snip] -->
/**
* Flags for H5Oget_native_info(). Theses flags determine which fields will be
* filled in in the \ref H5O_native_info_t struct.
*/
//! [H5O_native_info_fields_snip]
#define H5O_NATIVE_INFO_HDR 0x0008u /* Fill in the hdr field */
#define H5O_NATIVE_INFO_META_SIZE 0x0010u /* Fill in the meta_size field */
#define H5O_NATIVE_INFO_ALL (H5O_NATIVE_INFO_HDR | H5O_NATIVE_INFO_META_SIZE)
//! [H5O_native_info_fields_snip]
//! <!-- [H5O_native_info_fields_snip] -->
/* Convenience macro to check if the token is the 'undefined' token value */
#define H5O_IS_TOKEN_UNDEF(token) (!HDmemcmp(&(token), &(H5O_TOKEN_UNDEF), sizeof(H5O_token_t)))
@@ -105,46 +103,48 @@
/* Public Typedefs */
/*******************/
//! [H5O_type_t_snip]
/* Types of objects in file */
//! <!-- [H5O_type_t_snip] -->
/**
* Types of objects in file
*/
typedef enum H5O_type_t {
H5O_TYPE_UNKNOWN = -1, /* Unknown object type */
H5O_TYPE_GROUP, /* Object is a group */
H5O_TYPE_DATASET, /* Object is a dataset */
H5O_TYPE_NAMED_DATATYPE, /* Object is a named data type */
H5O_TYPE_MAP, /* Object is a map */
H5O_TYPE_NTYPES /* Number of different object types (must be last!) */
H5O_TYPE_UNKNOWN = -1, /**< Unknown object type */
H5O_TYPE_GROUP, /**< Object is a group */
H5O_TYPE_DATASET, /**< Object is a dataset */
H5O_TYPE_NAMED_DATATYPE, /**< Object is a named data type */
H5O_TYPE_MAP, /**< Object is a map */
H5O_TYPE_NTYPES /**< Number of different object types (must be last!) */
} H5O_type_t;
//! <!-- [H5O_type_t_snip] -->
//! [H5O_type_t_snip]
/* Information struct for object header metadata (for H5Oget_info/H5Oget_info_by_name/H5Oget_info_by_idx) */
//! [H5O_hdr_info_t_snip]
//! <!-- [H5O_hdr_info_t_snip] -->
/**
* Information struct for object header metadata (for
* H5Oget_info(), H5Oget_info_by_name(), H5Oget_info_by_idx())
*/
typedef struct H5O_hdr_info_t {
unsigned version; /* Version number of header format in file */
unsigned nmesgs; /* Number of object header messages */
unsigned nchunks; /* Number of object header chunks */
unsigned flags; /* Object header status flags */
unsigned version; /**< Version number of header format in file */
unsigned nmesgs; /**< Number of object header messages */
unsigned nchunks; /**< Number of object header chunks */
unsigned flags; /**< Object header status flags */
struct {
hsize_t total; /* Total space for storing object header in file */
hsize_t meta; /* Space within header for object header metadata information */
hsize_t mesg; /* Space within header for actual message information */
hsize_t free; /* Free space within object header */
hsize_t total; /**< Total space for storing object header in file */
hsize_t meta; /**< Space within header for object header metadata information */
hsize_t mesg; /**< Space within header for actual message information */
hsize_t free; /**< Free space within object header */
} space;
struct {
uint64_t present; /* Flags to indicate presence of message type in header */
uint64_t shared; /* Flags to indicate message type is shared in header */
uint64_t present; /**< Flags to indicate presence of message type in header */
uint64_t shared; /**< Flags to indicate message type is shared in header */
} mesg;
} H5O_hdr_info_t;
//! <!-- [H5O_hdr_info_t_snip] -->
//! [H5O_hdr_info_t_snip]
//! [H5O_info2_t_snip]
/* Data model information struct for objects */
/* (For H5Oget_info / H5Oget_info_by_name / H5Oget_info_by_idx version 3) */
//! <!-- [H5O_info2_t_snip] -->
/**
* Data model information struct for objects
* (For H5Oget_info(), H5Oget_info_by_name(), H5Oget_info_by_idx() version 3)
*/
typedef struct H5O_info2_t {
unsigned long fileno; /* File number that object is located in */
H5O_token_t token; /* Token representing the object */
@@ -156,44 +156,52 @@ typedef struct H5O_info2_t {
time_t btime; /* Birth time */
hsize_t num_attrs; /* # of attributes attached to object */
} H5O_info2_t;
//! <!-- [H5O_info2_t_snip] -->
//! [H5O_info2_t_snip]
//! [H5O_native_info_t_snip]
/* Native file format information struct for objects */
/* (For H5Oget_native_info / H5Oget_native_info_by_name / H5Oget_native_info_by_idx) */
//! <!-- [H5O_native_info_t_snip] -->
/**
* Native file format information struct for objects.
* (For H5Oget_native_info(), H5Oget_native_info_by_name(), H5Oget_native_info_by_idx())
*/
typedef struct H5O_native_info_t {
H5O_hdr_info_t hdr; /* Object header information */
H5O_hdr_info_t hdr; /**< Object header information */
/* Extra metadata storage for obj & attributes */
struct {
H5_ih_info_t obj; /* v1/v2 B-tree & local/fractal heap for groups, B-tree for chunked datasets */
H5_ih_info_t attr; /* v2 B-tree & heap for attributes */
H5_ih_info_t obj; /**< v1/v2 B-tree & local/fractal heap for groups, B-tree for chunked datasets */
H5_ih_info_t attr; /**< v2 B-tree & heap for attributes */
} meta_size;
} H5O_native_info_t;
//! <!-- [H5O_native_info_t_snip] -->
//! [H5O_native_info_t_snip]
/* Typedef for message creation indexes */
/**
* Typedef for message creation indexes
*/
typedef uint32_t H5O_msg_crt_idx_t;
/* Prototype for H5Ovisit/H5Ovisit_by_name() operator (version 3) */
//! [H5O_iterate2_t_snip]
//! <!-- [H5O_iterate2_t_snip] -->
/**
* Prototype for H5Ovisit(), H5Ovisit_by_name() operator (version 3)
*/
typedef herr_t (*H5O_iterate2_t)(hid_t obj, const char *name, const H5O_info2_t *info, void *op_data);
//! <!-- [H5O_iterate2_t_snip] -->
//! [H5O_iterate2_t_snip]
//! <!-- [H5O_mcdt_search_ret_t_snip] -->
typedef enum H5O_mcdt_search_ret_t {
H5O_MCDT_SEARCH_ERROR = -1, /* Abort H5Ocopy */
H5O_MCDT_SEARCH_CONT, /* Continue the global search of all committed datatypes in the destination file */
H5O_MCDT_SEARCH_STOP /* Stop the search, but continue copying. The committed datatype will be copied but
not merged. */
H5O_MCDT_SEARCH_ERROR = -1, /**< Abort H5Ocopy */
H5O_MCDT_SEARCH_CONT, /**< Continue the global search of all committed datatypes in the destination file
*/
H5O_MCDT_SEARCH_STOP /**< Stop the search, but continue copying. The committed datatype will be copied
but not merged. */
} H5O_mcdt_search_ret_t;
//! <!-- [H5O_mcdt_search_ret_t_snip] -->
/* Callback to invoke when completing the search for a matching committed datatype from the committed dtype
* list */
//! <!-- [H5O_mcdt_search_cb_t_snip] -->
/**
* Callback to invoke when completing the search for a matching committed
* datatype from the committed dtype list
*/
typedef H5O_mcdt_search_ret_t (*H5O_mcdt_search_cb_t)(void *op_data);
//! <!-- [H5O_mcdt_search_cb_t_snip] -->
/********************/
/* Public Variables */
@@ -249,6 +257,11 @@ extern "C" {
*
*/
H5_DLL hid_t H5Oopen(hid_t loc_id, const char *name, hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Oopen}
*/
H5_DLL hid_t H5Oopen_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *name, hid_t lapl_id, hid_t es_id);
@@ -332,6 +345,11 @@ H5_DLL hid_t H5Oopen_by_token(hid_t loc_id, H5O_token_t token);
*/
H5_DLL hid_t H5Oopen_by_idx(hid_t loc_id, const char *group_name, H5_index_t idx_type, H5_iter_order_t order,
hsize_t n, hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Oopen_by_idx}
*/
H5_DLL hid_t H5Oopen_by_idx_async(const char *app_file, const char *app_func, unsigned app_line, hid_t loc_id,
const char *group_name, H5_index_t idx_type, H5_iter_order_t order,
hsize_t n, hid_t lapl_id, hid_t es_id);
@@ -553,6 +571,11 @@ H5_DLL herr_t H5Oget_info3(hid_t loc_id, H5O_info2_t *oinfo, unsigned fields);
*/
H5_DLL herr_t H5Oget_info_by_name3(hid_t loc_id, const char *name, H5O_info2_t *oinfo, unsigned fields,
hid_t lapl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Oget_info_by_name}
*/
H5_DLL herr_t H5Oget_info_by_name_async(const char *app_file, const char *app_func, unsigned app_line,
hid_t loc_id, const char *name, H5O_info2_t *oinfo /*out*/,
unsigned fields, hid_t lapl_id, hid_t es_id);
@@ -968,7 +991,6 @@ H5_DLL herr_t H5Odecr_refcount(hid_t object_id);
* - H5Pset_copy_object()
* - H5Pset_create_intermediate_group()
* - H5Pset_mcdt_search_cb()
* .
* - Copying Committed Datatypes with #H5Ocopy - A comprehensive
* discussion of copying committed datatypes (PDF) in
* Advanced Topics in HDF5
@@ -980,6 +1002,11 @@ H5_DLL herr_t H5Odecr_refcount(hid_t object_id);
*/
H5_DLL herr_t H5Ocopy(hid_t src_loc_id, const char *src_name, hid_t dst_loc_id, const char *dst_name,
hid_t ocpypl_id, hid_t lcpl_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Ocopy}
*/
H5_DLL herr_t H5Ocopy_async(const char *app_file, const char *app_func, unsigned app_line, hid_t src_loc_id,
const char *src_name, hid_t dst_loc_id, const char *dst_name, hid_t ocpypl_id,
hid_t lcpl_id, hid_t es_id);
@@ -1522,6 +1549,11 @@ H5_DLL herr_t H5Ovisit_by_name3(hid_t loc_id, const char *obj_name, H5_index_t i
*
*/
H5_DLL herr_t H5Oclose(hid_t object_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Oclose}
*/
H5_DLL herr_t H5Oclose_async(const char *app_file, const char *app_func, unsigned app_line, hid_t object_id,
hid_t es_id);
@@ -1572,6 +1604,11 @@ H5_DLL herr_t H5Oclose_async(const char *app_file, const char *app_func, unsigne
*
*/
H5_DLL herr_t H5Oflush(hid_t obj_id);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Oflush}
*/
H5_DLL herr_t H5Oflush_async(const char *app_file, const char *app_func, unsigned app_line, hid_t obj_id,
hid_t es_id);
/**
@@ -1599,6 +1636,11 @@ H5_DLL herr_t H5Oflush_async(const char *app_file, const char *app_func, unsigne
*
*/
H5_DLL herr_t H5Orefresh(hid_t oid);
/**
* --------------------------------------------------------------------------
* \ingroup ASYNC
* \async_variant_of{H5Orefresh}
*/
H5_DLL herr_t H5Orefresh_async(const char *app_file, const char *app_func, unsigned app_line, hid_t oid,
hid_t es_id);
@@ -1873,46 +1915,49 @@ H5_DLLVAR const H5O_token_t H5O_TOKEN_UNDEF_g;
/* Typedefs */
/* A struct that's part of the H5G_stat_t structure (deprecated) */
//! [H5O_stat_t_snip]
//! <!-- [H5O_stat_t_snip] -->
/**
* A struct that's part of the \ref H5G_stat_t structure
* \deprecated
*/
typedef struct H5O_stat_t {
hsize_t size; /* Total size of object header in file */
hsize_t free; /* Free space within object header */
unsigned nmesgs; /* Number of object header messages */
unsigned nchunks; /* Number of object header chunks */
hsize_t size; /**< Total size of object header in file */
hsize_t free; /**< Free space within object header */
unsigned nmesgs; /**< Number of object header messages */
unsigned nchunks; /**< Number of object header chunks */
} H5O_stat_t;
//! [H5O_stat_t_snip]
//! <!-- [H5O_stat_t_snip] -->
//! [H5O_info1_t_snip]
/* Information struct for object */
/* (For H5Oget_info/H5Oget_info_by_name/H5Oget_info_by_idx versions 1 & 2) */
//! <!-- [H5O_info1_t_snip] -->
/**
* Information struct for object (For H5Oget_info(), H5Oget_info_by_name(),
* H5Oget_info_by_idx() versions 1 & 2.)
*/
typedef struct H5O_info1_t {
unsigned long fileno; /* File number that object is located in */
haddr_t addr; /* Object address in file */
H5O_type_t type; /* Basic object type (group, dataset, etc.) */
unsigned rc; /* Reference count of object */
time_t atime; /* Access time */
time_t mtime; /* Modification time */
time_t ctime; /* Change time */
time_t btime; /* Birth time */
hsize_t num_attrs; /* # of attributes attached to object */
H5O_hdr_info_t hdr; /* Object header information */
unsigned long fileno; /**< File number that object is located in */
haddr_t addr; /**< Object address in file */
H5O_type_t type; /**< Basic object type (group, dataset, etc.) */
unsigned rc; /**< Reference count of object */
time_t atime; /**< Access time */
time_t mtime; /**< Modification time */
time_t ctime; /**< Change time */
time_t btime; /**< Birth time */
hsize_t num_attrs; /**< # of attributes attached to object */
H5O_hdr_info_t hdr; /**< Object header information */
/* Extra metadata storage for obj & attributes */
struct {
H5_ih_info_t obj; /* v1/v2 B-tree & local/fractal heap for groups, B-tree for chunked datasets */
H5_ih_info_t attr; /* v2 B-tree & heap for attributes */
H5_ih_info_t obj; /**< v1/v2 B-tree & local/fractal heap for groups, B-tree for chunked datasets */
H5_ih_info_t attr; /**< v2 B-tree & heap for attributes */
} meta_size;
} H5O_info1_t;
//! <!-- [H5O_info1_t_snip] -->
//! [H5O_info1_t_snip]
/* Prototype for H5Ovisit/H5Ovisit_by_name() operator (versions 1 & 2) */
//! [H5O_iterate1_t_snip]
//! <!-- [H5O_iterate1_t_snip] -->
/**
* Prototype for H5Ovisit(), H5Ovisit_by_name() operator (versions 1 & 2)
*/
typedef herr_t (*H5O_iterate1_t)(hid_t obj, const char *name, const H5O_info1_t *info, void *op_data);
//! [H5O_iterate1_t_snip]
//! <!-- [H5O_iterate1_t_snip] -->
/* Function prototypes */
src/H5PLpublic.h
+2 -4
View File
@@ -28,8 +28,7 @@
*/
#define H5PL_NO_PLUGIN "::"
//! [H5PL_type_t_snip]
//! <!-- [H5PL_type_t_snip] -->
/**
* Plugin type (bit-position) used by the plugin library
*/
@@ -39,8 +38,7 @@ typedef enum H5PL_type_t {
H5PL_TYPE_VOL = 1, /**< VOL driver */
H5PL_TYPE_NONE = 2 /**< Sentinel: This must be last! */
} H5PL_type_t;
//! [H5PL_type_t_snip]
//! <!-- [H5PL_type_t_snip] -->
/* Common dynamic plugin type flags used by the set/get_loading_state functions */
#define H5PL_FILTER_PLUGIN 0x0001
src/H5Pmodule.h
+4 -1
View File
@@ -44,7 +44,6 @@
* and compressed.
*
* \todo Describe concisely what the functions in this module are about.
* \todo Clicking on "more" after "Property List Interface" at the top does not work
*
* \defgroup GPLO General Property List Operations
* \ingroup H5P
@@ -70,6 +69,10 @@
* \ingroup H5P
* \defgroup OCPPL Object Copy Properties
* \ingroup H5P
* \defgroup GACPL General Access Properties
* \ingroup H5P
* \defgroup MAPL Map Access Properties
* \ingroup H5P
*/
#endif /* H5Pmodule_H */
src/H5Ppublic.h
+2938 -192
View File
@@ -111,23 +111,50 @@ extern "C" {
/*******************/
/* Define property list class callback function pointer types */
//! [H5P_cls_create_func_t_snip]
//! <!-- [H5P_cls_create_func_t_snip] -->
typedef herr_t (*H5P_cls_create_func_t)(hid_t prop_id, void *create_data);
//! [H5P_cls_create_func_t_snip]
//! [H5P_cls_copy_func_t_snip]
//! <!-- [H5P_cls_create_func_t_snip] -->
//! <!-- [H5P_cls_copy_func_t_snip] -->
typedef herr_t (*H5P_cls_copy_func_t)(hid_t new_prop_id, hid_t old_prop_id, void *copy_data);
//! [H5P_cls_copy_func_t_snip]
//! [H5P_cls_close_func_t_snip]
//! <!-- [H5P_cls_copy_func_t_snip] -->
//! <!-- [H5P_cls_close_func_t_snip] -->
typedef herr_t (*H5P_cls_close_func_t)(hid_t prop_id, void *close_data);
//! [H5P_cls_close_func_t_snip]
//! <!-- [H5P_cls_close_func_t_snip] -->
/* Define property list callback function pointer types */
//! [H5P_prp_cb1_t_snip]
//! <!-- [H5P_prp_cb1_t_snip] -->
/**
* \brief Callback function for H5Pregister2(),H5Pregister1(),H5Pinsert2(),H5Pinsert1()
*
* \param[in] name The name of the property
* \param[in] size The size of the property in bytes
* \param[in,out] value The value for the property
* \return \herr_t
*
* \details The H5P_prp_cb1_t() describes the parameters used by the
* property create,copy and close callback functions.
*/
typedef herr_t (*H5P_prp_cb1_t)(const char *name, size_t size, void *value);
//! [H5P_prp_cb1_t_snip]
//! [H5P_prp_cb2_t_snip]
//! <!-- [H5P_prp_cb1_t_snip] -->
//! <!-- [H5P_prp_cb2_t_snip] -->
/**
* \brief Callback function for H5Pregister2(),H5Pregister1(),H5Pinsert2(),H5Pinsert1()
*
* \plist_id{prop_id}
* \param[in] name The name of the property
* \param[in] size The size of the property in bytes
* \param[in] value The value for the property
* \return \herr_t
*
* \details The H5P_prp_cb2_t() describes the parameters used by the
* property set ,copy and delete callback functions.
*/
typedef herr_t (*H5P_prp_cb2_t)(hid_t prop_id, const char *name, size_t size, void *value);
//! [H5P_prp_cb2_t_snip]
//! <!-- [H5P_prp_cb2_t_snip] -->
typedef H5P_prp_cb1_t H5P_prp_create_func_t;
typedef H5P_prp_cb2_t H5P_prp_set_func_t;
typedef H5P_prp_cb2_t H5P_prp_get_func_t;
@@ -135,60 +162,93 @@ typedef herr_t (*H5P_prp_encode_func_t)(const void *value, void **buf, size_t *s
typedef herr_t (*H5P_prp_decode_func_t)(const void **buf, void *value);
typedef H5P_prp_cb2_t H5P_prp_delete_func_t;
typedef H5P_prp_cb1_t H5P_prp_copy_func_t;
//! [H5P_prp_compare_func_t_snip]
//! <!-- [H5P_prp_compare_func_t_snip] -->
typedef int (*H5P_prp_compare_func_t)(const void *value1, const void *value2, size_t size);
//! [H5P_prp_compare_func_t_snip]
//! <!-- [H5P_prp_compare_func_t_snip] -->
typedef H5P_prp_cb1_t H5P_prp_close_func_t;
/* Define property list iteration function type */
//! [H5P_iterate_t_snip]
//! <!-- [H5P_iterate_t_snip] -->
typedef herr_t (*H5P_iterate_t)(hid_t id, const char *name, void *iter_data);
//! [H5P_iterate_t_snip]
//! <!-- [H5P_iterate_t_snip] -->
/* Actual IO mode property */
//! <!--[H5D_mpio_actual_chunk_opt_mode_t_snip] -->
/**
* Actual IO mode property
*
* \details The default value, #H5D_MPIO_NO_CHUNK_OPTIMIZATION, is used for all
* I/O operations that do not use chunk optimizations, including
* non-collective I/O and contiguous collective I/O.
*/
typedef enum H5D_mpio_actual_chunk_opt_mode_t {
/* The default value, H5D_MPIO_NO_CHUNK_OPTIMIZATION, is used for all I/O
* operations that do not use chunk optimizations, including non-collective
* I/O and contiguous collective I/O.
*/
H5D_MPIO_NO_CHUNK_OPTIMIZATION = 0,
/**< No chunk optimization was performed. Either no collective I/O was
attempted or the dataset wasn't chunked. */
H5D_MPIO_LINK_CHUNK,
/**< Collective I/O is performed on all chunks simultaneously. */
H5D_MPIO_MULTI_CHUNK
/**< Each chunk was individually assigned collective or independent I/O based
on what fraction of processes access the chunk. If the fraction is greater
than the multi chunk ratio threshold, collective I/O is performed on that
chunk. The multi chunk ratio threshold can be set using
H5Pset_dxpl_mpio_chunk_opt_ratio(). The default value is 60%. */
} H5D_mpio_actual_chunk_opt_mode_t;
//! <!--[H5D_mpio_actual_chunk_opt_mode_t_snip] -->
//! <!-- [H5D_mpio_actual_io_mode_t_snip] -->
/**
* The following values are conveniently defined as a bit field so that
* we can switch from the default to independent or collective and then to
* mixed without having to check the original value.
*/
typedef enum H5D_mpio_actual_io_mode_t {
/* The following four values are conveniently defined as a bit field so that
* we can switch from the default to independent or collective and then to
* mixed without having to check the original value.
*
* NO_COLLECTIVE means that either collective I/O wasn't requested or that
* no I/O took place.
*
* CHUNK_INDEPENDENT means that collective I/O was requested, but the
* chunk optimization scheme chose independent I/O for each chunk.
*/
H5D_MPIO_NO_COLLECTIVE = 0x0,
H5D_MPIO_NO_COLLECTIVE = 0x0,
/**< No collective I/O was performed. Collective I/O was not requested or
collective I/O isn't possible on this dataset */
H5D_MPIO_CHUNK_INDEPENDENT = 0x1,
H5D_MPIO_CHUNK_COLLECTIVE = 0x2,
H5D_MPIO_CHUNK_MIXED = 0x1 | 0x2,
/* The contiguous case is separate from the bit field. */
/**< HDF5 performed one the chunk collective optimization schemes and each
chunk was accessed independently */
H5D_MPIO_CHUNK_COLLECTIVE = 0x2,
/**< HDF5 performed one the chunk collective optimization schemes and each
chunk was accessed collectively */
H5D_MPIO_CHUNK_MIXED = 0x1 | 0x2,
/**< HDF5 performed one the chunk collective optimization schemes and some
chunks were accessed independently, some collectively. */
/** \internal The contiguous case is separate from the bit field. */
H5D_MPIO_CONTIGUOUS_COLLECTIVE = 0x4
/**< Collective I/O was performed on a contiguous dataset */
} H5D_mpio_actual_io_mode_t;
//! <!-- [H5D_mpio_actual_io_mode_t_snip] -->
/* Broken collective IO property */
//! <!-- [H5D_mpio_no_collective_cause_t_snip] -->
/**
* Broken collective IO property
*/
typedef enum H5D_mpio_no_collective_cause_t {
H5D_MPIO_COLLECTIVE = 0x00,
H5D_MPIO_SET_INDEPENDENT = 0x01,
H5D_MPIO_DATATYPE_CONVERSION = 0x02,
H5D_MPIO_DATA_TRANSFORMS = 0x04,
H5D_MPIO_MPI_OPT_TYPES_ENV_VAR_DISABLED = 0x08,
H5D_MPIO_NOT_SIMPLE_OR_SCALAR_DATASPACES = 0x10,
H5D_MPIO_NOT_CONTIGUOUS_OR_CHUNKED_DATASET = 0x20,
H5D_MPIO_PARALLEL_FILTERED_WRITES_DISABLED = 0x40,
H5D_MPIO_COLLECTIVE = 0x00,
/**< Collective I/O was performed successfully */
H5D_MPIO_SET_INDEPENDENT = 0x01,
/**< Collective I/O was not performed because independent I/O was requested */
H5D_MPIO_DATATYPE_CONVERSION = 0x02,
/**< Collective I/O was not performed because datatype conversions were required */
H5D_MPIO_DATA_TRANSFORMS = 0x04,
/**< Collective I/O was not performed because data transforms needed to be applied */
H5D_MPIO_MPI_OPT_TYPES_ENV_VAR_DISABLED = 0x08,
/**< \todo FIXME! */
H5D_MPIO_NOT_SIMPLE_OR_SCALAR_DATASPACES = 0x10,
/**< Collective I/O was not performed because one of the dataspaces was neither simple nor scalar */
H5D_MPIO_NOT_CONTIGUOUS_OR_CHUNKED_DATASET = 0x20,
/**< Collective I/O was not performed because the dataset was neither contiguous nor chunked */
H5D_MPIO_PARALLEL_FILTERED_WRITES_DISABLED = 0x40,
/**< \todo FIXME! */
H5D_MPIO_ERROR_WHILE_CHECKING_COLLECTIVE_POSSIBLE = 0x80,
H5D_MPIO_NO_COLLECTIVE_MAX_CAUSE = 0x100
/**< \todo FIXME! */
H5D_MPIO_NO_COLLECTIVE_MAX_CAUSE = 0x100
/**< Sentinel */
} H5D_mpio_no_collective_cause_t;
//! <!-- [H5D_mpio_no_collective_cause_t_snip] -->
/********************/
/* Public Variables */
@@ -493,8 +553,6 @@ H5_DLL hid_t H5Pcreate(hid_t cls_id);
* list of this class is being created. The #H5P_cls_create_func_t
* callback function is defined as follows:
*
* \todo fix snippets to work, when you click on them.
*
* \snippet this H5P_cls_create_func_t_snip
*
* The parameters to this callback function are defined as follows:
@@ -1233,8 +1291,6 @@ H5_DLL herr_t H5Pget_size(hid_t id, const char *name, size_t *size);
* property list objects; the initial value is assumed to
* have any necessary setup already performed on it.
*
* \todo "cpp_note" goes here
*
* \since 1.8.0
*
*/
@@ -1326,9 +1382,6 @@ H5_DLL htri_t H5Pisa_class(hid_t plist_id, hid_t pclass_id);
* If the membership changes during the iteration, the function's
* behavior is undefined.
*
*
* \todo "cpp_note" goes here
*
* \since 1.4.0
*
*/
@@ -1582,16 +1635,12 @@ H5_DLL int H5Piterate(hid_t id, int *idx, H5P_iterate_t iter_func, void *iter_da
* property is being closed. The #H5P_prp_close_func_t callback
* function is defined as follows:
*
* \snippet this H5P_prp_cb2_t_snip
* \snippet this H5P_prp_cb1_t_snip
*
* The parameters to the callback function are defined as follows:
*
* <table>
* <tr>
* <td>\ref hid_t \c prop_id</td>
* <td>IN: The identifier of the property list being closed</td>
* </tr>
* <tr>
* <td>\Code{const char * name}</td>
* <td>IN: The name of the property in the list</td>
* </tr>
@@ -1611,8 +1660,6 @@ H5_DLL int H5Piterate(hid_t id, int *idx, H5P_iterate_t iter_func, void *iter_da
* list close routine returns an error value but the property list is
* still closed.
*
* \todo "cpp_note" goes here
*
* \since 1.8.0
*
*/
@@ -1701,7 +1748,7 @@ H5_DLL herr_t H5Punregister(hid_t pclass_id, const char *name);
/* Object creation property list (OCPL) routines */
/**
* \ingroup OCPL
* \ingroup DCPL
*
* \brief Verifies that all required filters are available
*
@@ -1805,7 +1852,7 @@ H5_DLL herr_t H5Pget_attr_phase_change(hid_t plist_id, unsigned *max_compact, un
* \todo Signature for H5Pget_filter2 is different in H5Pocpl.c than in
* H5Ppublic.h
*
* \plist_id{plist_id}
* \ocpl_id{plist_id}
* \param[in] idx Sequence number within the filter pipeline of the filter
* for which information is sought
* \param[out] flags Bit vector specifying certain general properties of the
@@ -1869,7 +1916,7 @@ H5_DLL H5Z_filter_t H5Pget_filter2(hid_t plist_id, unsigned idx, unsigned int *f
*
* \brief Returns information about the specified filter
*
* \plist_id
* \ocpl_id{plist_id}
* \param[in] filter_id Filter identifier
* \param[out] flags Bit vector specifying certain general
* properties of the filter
@@ -1927,10 +1974,7 @@ H5_DLL herr_t H5Pget_filter_by_id2(hid_t plist_id, H5Z_filter_t filter_id, unsig
*
* \brief Returns the number of filters in the pipeline
*
* \todo Signature for H5Pget_nfilters() is different in H5Pocpl.c than in
* H5Ppublic.h.
*
* \plist_id
* \ocpl_id{plist_id}
*
* \return Returns the number of filters in the pipeline if successful;
* otherwise returns a negative value.
@@ -1938,8 +1982,8 @@ H5_DLL herr_t H5Pget_filter_by_id2(hid_t plist_id, H5Z_filter_t filter_id, unsig
* \details H5Pget_nfilters() returns the number of filters defined in the
* filter pipeline associated with the property list \p plist_id.
*
* In each pipeline, the filters are numbered from 0 through N-1,
* where N is the value returned by this function. During output to
* In each pipeline, the filters are numbered from 0 through \Code{N-1},
* where \c N is the value returned by this function. During output to
* the file, the filters are applied in increasing order; during
* input from the file, they are applied in decreasing order.
*
@@ -1987,7 +2031,7 @@ H5_DLL herr_t H5Pget_obj_track_times(hid_t plist_id, hbool_t *track_times);
*
* \brief Modifies a filter in the filter pipeline
*
* \plist_id
* \ocpl_id{plist_id}
* \param[in] filter Filter to be modified
* \param[in] flags Bit vector specifying certain general properties
* of the filter
@@ -2016,7 +2060,7 @@ H5_DLL herr_t H5Pmodify_filter(hid_t plist_id, H5Z_filter_t filter, unsigned int
*
* \brief Delete one or more filters in the filter pipeline
*
* \plist_id
* \ocpl_id{plist_id}
* \param[in] filter Filter to be deleted
*
* \return \herr_t
@@ -2170,7 +2214,7 @@ H5_DLL herr_t H5Pset_attr_phase_change(hid_t plist_id, unsigned max_compact, uns
*
* \brief Sets deflate (GNU gzip) compression method and compression level
*
* \plist_id
* \ocpl_id{plist_id}
* \param[in] level Compression level
*
* \return \herr_t
@@ -2226,7 +2270,7 @@ H5_DLL herr_t H5Pset_deflate(hid_t plist_id, unsigned level);
*
* \brief Adds a filter to the filter pipeline
*
* \param[in] plist_id Dataset or group creation property list identifier
* \ocpl_id{plist_id}
* \param[in] filter Filter identifier for the filter to be added to the
* pipeline
* \param[in] flags Bit vector specifying certain general properties of
@@ -2478,7 +2522,7 @@ H5_DLL herr_t H5Pset_deflate(hid_t plist_id, unsigned level);
* (The SZIP filter is an exception to this rule; see H5Pset_szip()
* for details.)
*
* \todo Removed several references to links to documentation
* \see \ref_filter_pipe, \ref_group_impls
*
* \version 1.8.5 Function applied to group creation property lists.
* \since 1.6.0
@@ -2491,7 +2535,7 @@ H5_DLL herr_t H5Pset_filter(hid_t plist_id, H5Z_filter_t filter, unsigned int fl
*
* \brief Sets up use of the Fletcher32 checksum filter
*
* \param[in] plist_id Dataset or group creation property list identifier
* \ocpl_id{plist_id}
*
* \return \herr_t
*
@@ -2672,7 +2716,7 @@ H5_DLL herr_t H5Pget_shared_mesg_index(hid_t plist_id, unsigned index_num, unsig
/**
* \ingroup FCPL
*
* \brief Retrieves number of shared object header message indexes in file
* \brief Retrieves the number of shared object header message indexes in file
* creation property list
*
* \fcpl_id{plist_id}
@@ -3237,8 +3281,8 @@ H5_DLL herr_t H5Pget_core_write_tracking(hid_t fapl_id, hbool_t *is_enabled, siz
* <td>#H5FD_SEC2</td>
* <td>This driver uses POSIX file-system functions like read and
* write to perform I/O to a single, permanent file on local disk
* with no system buffering. This driver is POSIX-compliant and is
* the default file driver for all systems.</td>
* with no system buffering. This driver is POSIX-compliant and
* is the default file driver for all systems.</td>
* <td>H5Pset_fapl_sec2()</td>
* </tr>
* <tr>
@@ -3276,8 +3320,9 @@ H5_DLL herr_t H5Pget_core_write_tracking(hid_t fapl_id, hbool_t *is_enabled, siz
* <td>#H5FD_CORE</td>
* <td>With this driver, an application can work with a file in
* memory for faster reads and writes. File contents are kept in
* memory until the file is closed. At closing, the memory version
* of the file can be written back to disk or abandoned.</td>
* memory until the file is closed. At closing, the memory
* version of the file can be written back to disk or abandoned.
* </td>
* <td>H5Pset_fapl_core()</td>
* </tr>
* <tr>
@@ -3286,7 +3331,8 @@ H5_DLL herr_t H5Pget_core_write_tracking(hid_t fapl_id, hbool_t *is_enabled, siz
* <td>With this driver, the HDF5 file’s address space is partitioned
* into pieces and sent to separate storage files using an
* underlying driver of the user’s choice. This driver is for
* systems that do not support files larger than 2 gigabytes.</td>
* systems that do not support files larger than 2 gigabytes.
* </td>
* <td>H5Pset_fapl_family()</td>
* </tr>
* <tr>
@@ -3352,7 +3398,7 @@ H5_DLL hid_t H5Pget_driver(hid_t plist_id);
* struct. Driver-specific versions of that struct are defined
* for each low-level driver in the relevant source code file
* H5FD*.c. For example, the struct used for the MULTI driver is
* #H5FD_multi_fapl_t defined in H5FDmulti.c.
* \c H5FD_multi_fapl_t defined in H5FDmulti.c.
*
* If no driver-specific properties have been registered,
* H5Pget_driver_info() returns NULL.
@@ -3369,8 +3415,50 @@ H5_DLL hid_t H5Pget_driver(hid_t plist_id);
*
*/
H5_DLL const void *H5Pget_driver_info(hid_t plist_id);
H5_DLL herr_t H5Pget_elink_file_cache_size(hid_t plist_id, unsigned *efc_size);
H5_DLL herr_t H5Pget_evict_on_close(hid_t fapl_id, hbool_t *evict_on_close);
/**
* \ingroup FAPL
*
* \brief Retrieves the size of the external link open file cache
*
* \fapl_id{plist_id}
* \param[out] efc_size External link open file cache size in number of files
*
* \return \herr_t
*
* \details H5Pget_elink_file_cache_size() retrieves the number of files that
* can be held open in an external link open file cache.
*
* \since 1.8.7
*
*/
H5_DLL herr_t H5Pget_elink_file_cache_size(hid_t plist_id, unsigned *efc_size);
/**
* \ingroup FAPL
*
* \brief Retrieves the file access property list setting that determines
* whether an HDF5 object will be evicted from the library's metadata
* cache when it is closed
*
* \fapl_id
* \param[out] evict_on_close Pointer to a variable that will indicate if
* the object will be evicted on close
*
* \return \herr_t
*
* \details The library's metadata cache is fairly conservative about holding on
* to HDF5 object metadata (object headers, chunk index structures,
* etc.), which can cause the cache size to grow, resulting in memory
* pressure on an application or system. When enabled, the "evict on
* close" property will cause all metadata for an object to be
* immediately evicted from the cache as long as it is not referenced
* by any other open object.
*
* See H5Pset_evict_on_close() for additional notes on behavior.
*
* \since 1.10.1
*
*/
H5_DLL herr_t H5Pget_evict_on_close(hid_t fapl_id, hbool_t *evict_on_close);
/**
* \ingroup FAPL
*
@@ -3386,9 +3474,7 @@ H5_DLL herr_t H5Pget_evict_on_close(hid_t fapl_id, hbool_t *evict_on_close)
* application can retrieve a file handle for low-level access to
* a particular member of a family of files. The file handle is
* retrieved with a separate call to H5Fget_vfd_handle() (or,
* in special circumstances, to H5FDget_vfd_handle()).
*
* \todo References the VFL documentation.
* in special circumstances, to H5FDget_vfd_handle(), see \ref VFL).
*
* \since 1.6.0
*
@@ -3415,8 +3501,113 @@ H5_DLL herr_t H5Pget_family_offset(hid_t fapl_id, hsize_t *offset);
*
*/
H5_DLL herr_t H5Pget_fclose_degree(hid_t fapl_id, H5F_close_degree_t *degree);
/**
* \ingroup FAPL
*
* \brief Retrieves a copy of the file image designated as the initial content
* and structure of a file
*
* \fapl_id
* \param[in,out] buf_ptr_ptr On input, \c NULL or a pointer to a
* pointer to a buffer that contains the
* file image.\n On successful return, if \p buf_ptr_ptr is not
* \c NULL, \Code{*buf_ptr_ptr} will contain a pointer to a copy
* of the initial image provided in the last call to
* H5Pset_file_image() for the supplied \p fapl_id. If no initial
* image has been set, \Code{*buf_ptr_ptr} will be \c NULL.
* \param[in,out] buf_len_ptr On input, \c NULL or a pointer to a buffer
* specifying the required size of the buffer to hold the file
* image.\n On successful return, if \p buf_len_ptr was not
* passed in as \c NULL, \p buf_len_ptr will return the required
* size in bytes of the buffer to hold the initial file image in
* the supplied file access property list, \p fapl_id. If no
* initial image is set, the value of \Code{*buf_len_ptr} will be
* set to 0 (zero)
* \return \herr_t
*
* \details H5Pget_file_image() allows an application to retrieve a copy of the
* file image designated for a VFD to use as the initial contents of a file.
*
* If file image callbacks are defined, H5Pget_file_image() will use
* them when allocating and loading the buffer to return to the
* application (see H5Pset_file_image_callbacks()). If file image
* callbacks are not defined, the function will use \c malloc and \c
* memcpy. When \c malloc and \c memcpy are used, it is the caller’s
* responsibility to discard the returned buffer with a call to \c
* free.
*
* It is the responsibility of the calling application to free the
* buffer whose address is returned in \p buf_ptr_ptr. This can be
* accomplished with \c free if file image callbacks have not been set
* (see H5Pset_file_image_callbacks()) or with the appropriate method
* if file image callbacks have been set.
*
* \see H5LTopen_file_image(), H5Fget_file_image(), H5Pset_file_image(),
* H5Pset_file_image_callbacks(), H5Pget_file_image_callbacks(),
* \ref H5FD_file_image_callbacks_t, \ref H5FD_file_image_op_t,
* <a href="https://portal.hdfgroup.org/display/HDF5/HDF5+File+Image+Operations">
* HDF5 File Image Operations</a>.
*
*
* \since 1.8.9
*
*/
H5_DLL herr_t H5Pget_file_image(hid_t fapl_id, void **buf_ptr_ptr, size_t *buf_len_ptr);
/**
* \ingroup FAPL
*
* \brief Retrieves callback routines for working with file images
*
* \fapl_id
* \param[in,out] callbacks_ptr Pointer to the instance of the
* #H5FD_file_image_callbacks_t struct in which the callback
* routines are to be returned\n
* Struct fields must be initialized to NULL before the call
* is made.\n
* Struct field contents upon return will match those passed in
* in the last H5Pset_file_image_callbacks() call for the file
* access property list \p fapl_id.
* \return \herr_t
*
* \details H5Pget_file_image_callbacks() retrieves the callback routines set for
* working with file images opened with the file access property list
* \p fapl_id.
*
* The callbacks must have been previously set with
* H5Pset_file_image_callbacks() in the file access property list.
*
* Upon the successful return of H5Pset_file_image_callbacks(), the
* fields in the instance of the #H5FD_file_image_callbacks_t struct
* pointed to by \p callbacks_ptr will contain the same values as were
* passed in the most recent H5Pset_file_image_callbacks() call for the
* file access property list \p fapl_id.
*
* \see H5LTopen_file_image(), H5Fget_file_image(), H5Pset_file_image(),
* H5Pset_file_image_callbacks(), H5Pget_file_image_callbacks(),
* \ref H5FD_file_image_callbacks_t, \ref H5FD_file_image_op_t,
* <a href="https://portal.hdfgroup.org/display/HDF5/HDF5+File+Image+Operations">
* HDF5 File Image Operations</a>.
*
* \since 1.8.9
*
*/
H5_DLL herr_t H5Pget_file_image_callbacks(hid_t fapl_id, H5FD_file_image_callbacks_t *callbacks_ptr);
/**
* \ingroup FAPL
*
* \brief Retrieves the file locking property values
*
* \fapl_id
* \param[out] use_file_locking File locking flag
* \param[out] ignore_when_disabled Ignore when disabled flag
* \return \herr_t
*
* \details H5Pget_file_locking() retrieves the file locking property values for
* the file access property list specified by \p fapl_id.
*
* \since 1.10.7
*
*/
H5_DLL herr_t H5Pget_file_locking(hid_t fapl_id, hbool_t *use_file_locking, hbool_t *ignore_when_disabled);
/**
* \ingroup FAPL
@@ -3471,11 +3662,212 @@ H5_DLL herr_t H5Pget_gc_references(hid_t fapl_id, unsigned *gc_ref /*out*/);
*
*/
H5_DLL herr_t H5Pget_libver_bounds(hid_t plist_id, H5F_libver_t *low, H5F_libver_t *high);
/**
* \ingroup FAPL
*
* \brief Get the current initial metadata cache configuration from the
* provided file access property list
*
* \fapl_id{plist_id}
* \param[in,out] config_ptr Pointer to the instance of #H5AC_cache_config_t
* in which the current metadata cache configuration is to be
* reported
* \return \herr_t
*
* \note The \c in direction applies only to the \ref H5AC_cache_config_t.version
* field. All other fields are \c out parameters.
*
* \details The fields of the #H5AC_cache_config_t structure are shown
* below:
* \snippet H5ACpublic.h H5AC_cache_config_t_snip
* \click4more
*
* H5Pget_mdc_config() gets the initial metadata cache configuration
* contained in a file access property list and loads it into the
* instance of #H5AC_cache_config_t pointed to by the \p config_ptr
* parameter. This configuration is used when the file is opened.
*
* Note that the version field of \Code{*config_ptr} must be
* initialized; this allows the library to support earlier versions of
* the #H5AC_cache_config_t structure.
*
* See the overview of the metadata cache in the special topics section
* of the user guide for details on the configuration data returned. If
* you haven't read and understood that documentation, the results of
* this call will not make much sense.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pget_mdc_config(hid_t plist_id, H5AC_cache_config_t *config_ptr); /* out */
/**
* \ingroup FAPL
*
* \brief Retrieves the metadata cache image configuration values for a file
* access property list
*
* \fapl_id{plist_id}
* \param[out] config_ptr Pointer to metadata cache image configuration values
* \return \herr_t
*
* \details H5Pget_mdc_image_config() retrieves the metadata cache image values
* into \p config_ptr for the file access property list specified in \p
* plist_id.
*
* #H5AC_cache_image_config_t is defined as follows:
* \snippet H5ACpublic.h H5AC_cache_image_config_t_snip
* \click4more
*
* \since 1.10.1
*/
H5_DLL herr_t H5Pget_mdc_image_config(hid_t plist_id, H5AC_cache_image_config_t *config_ptr /*out*/);
/**
* \ingroup FAPL
*
* \brief Gets metadata cache logging options
*
* \fapl_id{plist_id}
* \param[out] is_enabled Flag whether logging is enabled
* \param[out] location Location of log in UTF-8/ASCII (file path/name) (On
* Windows, this must be ASCII)
* \param[out] location_size Size in bytes of the location string
* \param[out] start_on_access Whether the logging begins as soon as the file is
* opened or created
* \return \herr_t
*
* \details The metadata cache is a central part of the HDF5 library through
* which all file metadata reads and writes take place. File metadata
* is normally invisible to the user and is used by the library for
* purposes such as locating and indexing data. File metadata should
* not be confused with user metadata, which consists of attributes
* created by users and attached to HDF5 objects such as datasets via
* \ref H5A API calls.
*
* Due to the complexity of the cache, a trace/logging feature has been
* created that can be used by HDF5 developers for debugging and
* performance analysis. The functions that control this functionality
* will normally be of use to a very limited number of developers
* outside of The HDF Group. The functions have been documented to help
* users create logs that can be sent with bug reports.
*
* Control of the log functionality is straightforward. Logging is
* enabled via the H5Pset_mdc_log_options() function, which will modify
* the file access property list used to open or create a file. This
* function has a flag that determines whether logging begins at file
* open or starts in a paused state. Log messages can then be
* controlled via the H5Fstart_mdc_logging() / H5Fstop_mdc_logging()
* functions. H5Pget_mdc_log_options() can be used to examine a file
* access property list, and H5Fget_mdc_logging_status() will return
* the current state of the logging flags.
*
* The log format is described in the
* <a href="https://bit.ly/2PG6fNv">Metadata Cache Logging</a> document.
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pget_mdc_log_options(hid_t plist_id, hbool_t *is_enabled, char *location,
size_t *location_size, hbool_t *start_on_access);
H5_DLL herr_t H5Pget_meta_block_size(hid_t fapl_id, hsize_t *size /*out*/);
/**
* \ingroup FAPL
*
* \brief Returns the current metadata block size setting
*
* \fapl_id{fapl_id}
* \param[out] size Minimum size, in bytes, of metadata block allocations
*
* \return \herr_t
*
* \details Returns the current minimum size, in bytes, of new
* metadata block allocations. This setting is retrieved from the
* file access property list \p fapl_id.
*
* This value is set by H5Pset_meta_block_size() and is
* retrieved from the file access property list \p fapl_id.
*
* \since 1.4.0
*/
H5_DLL herr_t H5Pget_meta_block_size(hid_t fapl_id, hsize_t *size);
/**
* \ingroup FAPL
*
* \brief Retrieves the number of read attempts from a file access
* property list
*
* \fapl_id{plist_id}
* \param[out] attempts The number of read attempts
*
* \return \herr_t
*
* \details H5Pget_metadata_read_attempts() retrieves the number of read
* attempts that is set in the file access property list \p plist_id.
*
* For a default file access property list, the value retrieved
* will depend on whether the user sets the number of attempts via
* H5Pset_metadata_read_attempts():
*
* <ul>
*
* <li>If the number of attempts is set to N, the value
* returned will be N.
* <li>If the number of attempts is not set, the value returned
* will be the default for non-SWMR access (1). SWMR is short
* for single-writer/multiple-reader.
* </ul>
*
* For the file access property list of a specified HDF5 file,
* the value retrieved will depend on how the file is opened
* and whether the user sets the number of read attempts via
* H5Pset_metadata_read_attempts():
*
* <ul>
* <li>For a file opened with SWMR access:
*
* <ul>
* <li> If the number of attempts is set to N, the value
* returned will be N.
* <li> If the number of attempts is not set, the value
* returned will be the default for SWMR access (100).
* </ul>
* <li>For a file opened without SWMR access, the value
* retrieved will always be the default for non-SWMR access
* (1). The value set via H5Pset_metadata_read_attempts() does
* not have any effect on non-SWMR access.
* </ul>
*
* \par Failure Modes
* \parblock
*
* When the input property list is not a file access property list.
*
* When the library is unable to retrieve the number of read attempts from
* the file access property list.
*
* \endparblock
*
* \par Examples
* \parblock
*
* The first example illustrates the two cases for retrieving the number
* of read attempts from a default file access property list.
*
* \include H5Pget_metadata_read_attempts.1.c
*
* The second example illustrates the two cases for retrieving the
* number of read attempts from the file access property list of a file
* opened with SWMR acccess.
*
* \include H5Pget_metadata_read_attempts.2.c
*
* The third example illustrates the two cases for retrieving the number
* of read attempts from the file access property list of a file opened
* with non-SWMR acccess.
*
* \include H5Pget_metadata_read_attempts.3.c
*
* \endparblock
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pget_metadata_read_attempts(hid_t plist_id, unsigned *attempts);
/**
* \ingroup FAPL
@@ -3510,10 +3902,102 @@ H5_DLL herr_t H5Pget_metadata_read_attempts(hid_t plist_id, unsigned *attempts);
*
*/
H5_DLL herr_t H5Pget_multi_type(hid_t fapl_id, H5FD_mem_t *type);
/**
* \ingroup FAPL
*
* \brief Retrieves the object flush property values from the file access property list
*
* \fapl_id{plist_id}
* \param[in] func The user-defined callback function
* \param[in] udata The user-defined input data for the callback function
*
* \return \herr_t
*
* \details H5Pget_object_flush_cb() gets the user-defined callback
* function that is set in the file access property list
* \p fapl_id and stored in the parameter \p func. The callback is
* invoked whenever an object flush occurs in the file. This
* routine also obtains the user-defined input data that is
* passed along to the callback function in the parameter
* \p udata.
*
* \par Example
* \parblock
* The example below illustrates the usage of this routine to obtain the
* object flush property values.
*
* \include H5Pget_object_flush_cb.c
* \endparblock
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pget_object_flush_cb(hid_t plist_id, H5F_flush_cb_t *func, void **udata);
H5_DLL herr_t H5Pget_page_buffer_size(hid_t plist_id, size_t *buf_size, unsigned *min_meta_per,
unsigned *min_raw_per);
/**
* \ingroup FAPL
*
* \brief Retrieves the maximum size for the page buffer and the minimum
percentage for metadata and raw data pages
*
* \fapl_id{plist_id}
* \param[out] buf_size Maximum size, in bytes, of the page buffer
* \param[out] min_meta_perc Minimum metadata percentage to keep in the
* page buffer before allowing pages containing metadata to
* be evicted
*
* \param[out] min_raw_perc Minimum raw data percentage to keep in the
* page buffer before allowing pages containing raw data to
* be evicted
*
* \return \herr_t
*
* \details H5Pget_page_buffer_size() retrieves \p buf_size, the maximum
* size in bytes of the page buffer, \p min_meta_perc, the
* minimum metadata percentage, and \p min_raw_perc, the
* minimum raw data percentage.
*
* \since 1.10.1
*/
H5_DLL herr_t H5Pget_page_buffer_size(hid_t plist_id, size_t *buf_size, unsigned *min_meta_perc,
unsigned *min_raw_perc);
/**
* \ingroup FAPL
*
* \brief Returns maximum data sieve buffer size
*
* \fapl_id{fapl_id}
* \param[in] size Maximum size, in bytes, of data sieve buffer
*
* \return \herr_t
*
* \details H5Pget_sieve_buf_size() retrieves, size, the current maximum
* size of the data sieve buffer.
*
* This value is set by H5Pset_sieve_buf_size() and is retrieved
* from the file access property list fapl_id.
*
* \version 1.6.0 The \p size parameter has changed from type \c hsize_t
* to \c size_t
* \since 1.4.0
*/
H5_DLL herr_t H5Pget_sieve_buf_size(hid_t fapl_id, size_t *size /*out*/);
/**
* \ingroup FAPL
*
* \brief Retrieves the current small data block size setting
*
* \fapl_id{fapl_id}
* \param[out] size Maximum size, in bytes, of the small data block
*
* \result \herr_t
*
* \details H5Pget_small_data_block_size() retrieves the current setting
* for the size of the small data block.
*
* If the returned value is zero (0), the small data block
* mechanism has been disabled for the file.
*
* \since 1.4.4
*/
H5_DLL herr_t H5Pget_small_data_block_size(hid_t fapl_id, hsize_t *size /*out*/);
/**
* \ingroup FAPL
@@ -3684,6 +4168,64 @@ H5_DLL herr_t H5Pset_alignment(hid_t fapl_id, hsize_t threshold, hsize_t alignme
*/
H5_DLL herr_t H5Pset_cache(hid_t plist_id, int mdc_nelmts, size_t rdcc_nslots, size_t rdcc_nbytes,
double rdcc_w0);
/**
* \ingroup FAPL
*
* \brief Sets write tracking information for core driver, #H5FD_CORE
*
* \fapl_id{fapl_id}
* \param[in] is_enabled Boolean value specifying whether feature is
enabled
* \param[in] page_size Positive integer specifying size, in bytes, of
* write aggregation pages Value of 1 (one) enables
* tracking with no paging.
*
* \return \herr_t
*
* \details When a file is created or opened for writing using the core
* virtual file driver (VFD) with the backing store option
* turned on, the core driver can be configured to track
* changes to the file and write out only the modified bytes.
*
* This write tracking feature is enabled and disabled with \p
* is_enabled. The default setting is that write tracking is
* disabled, or off.
*
* To avoid a large number of small writes, changes can
* be aggregated into pages of a user-specified size, \p
* page_size.
*
* Setting \p page_size to 1 enables tracking with no page
* aggregation.
*
* The backing store option is set via the function
* H5Pset_fapl_core.
*
* \attention
* \parblock
* This function is only for use with the core VFD and must
* be used after the call to H5Pset_fapl_core(). It is an error
* to use this function with any other VFD.
*
* It is an error to use this function when the backing store
* flag has not been set using H5Pset_fapl_core().
*
* This function only applies to the backing store write
* operation which typically occurs when the file is flushed
* or closed. This function has no relationship to the
* increment parameter passed to H5Pset_fapl_core().
*
* For optimum performance, the \p page_size parameter should be
* a power of two.
*
* It is an error to set the page size to 0.
* \endparblock
*
* \version 1.8.14 C function modified in this release to return error
* if \p page_size is set to 0 (zero).
* \since 1.8.13
*
*/
H5_DLL herr_t H5Pset_core_write_tracking(hid_t fapl_id, hbool_t is_enabled, size_t page_size);
/**
* \ingroup FAPL
@@ -3712,8 +4254,141 @@ H5_DLL herr_t H5Pset_core_write_tracking(hid_t fapl_id, hbool_t is_enabled, size
*
*/
H5_DLL herr_t H5Pset_driver(hid_t plist_id, hid_t driver_id, const void *driver_info);
/**
* \ingroup FAPL
*
* \brief Sets the number of files that can be held open in an external
* link open file cache
*
* \par Motivation
* \parblock
* The <em>external link open file cache</em> holds files open after
* they have been accessed via an external link. This cache reduces
* the number of times such files are opened when external links are
* accessed repeatedly and can siginificantly improves performance in
* certain heavy-use situations and when low-level file opens or closes
* are expensive.
*
* H5Pset_elink_file_cache_size() sets the number of files
* that will be held open in an external link open file
* cache. H5Pget_elink_file_cache_size() retrieves the size of an existing
* cache; and H5Fclear_elink_file_cache() clears an existing cache without
* closing it.
* \endparblock
*
* \fapl_id{plist_id}
* \param[in] efc_size External link open file cache size in number of files
* <em>Default setting is 0 (zero).</em>
*
* \return \herr_t
*
* \details H5Pset_elink_file_cache_size() specifies the number of files
* that will be held open in an external link open file cache.
*
* The default external link open file cache size is 0 (zero),
* meaning that files accessed via an external link are not
* held open. Setting the cache size to a positive integer
* turns on the cache; setting the size back to zero turns it
* off.
*
* With this property set, files are placed in the external
* link open file cache cache when they are opened via an
* external link. Files are then held open until either
* they are evicted from the cache or the parent file is
* closed. This property setting can improve performance when
* external links are repeatedly accessed.
*
* When the cache is full, files will be evicted using a least
* recently used (LRU) scheme; the file which has gone the
* longest time without being accessed through the parent file
* will be evicted and closed if nothing else is holding that
* file open.
*
* Files opened through external links inherit the parent
* file’s file access property list by default, and therefore
* inherit the parent file’s external link open file cache
* setting.
*
* When child files contain external links of their own, the
* caches can form a graph of cached external files. Closing
* the last external reference to such a graph will recursively
* close all files in the graph, even if cycles are present.
* \par Example
* \parblock
* The following code sets up an external link open file cache that will
* hold open up to 8 files reached through external links:
*
* \code
* status = H5Pset_elink_file_cache_size(fapl_id, 8);
* \endcode
* \endparblock
*
* \since 1.8.7
*/
H5_DLL herr_t H5Pset_elink_file_cache_size(hid_t plist_id, unsigned efc_size);
/**
* \ingroup FAPL
*
* \brief Controls the library's behavior of evicting metadata associated with
* a closed object
*
* \fapl_id
* \param[in] evict_on_close Whether the HDF5 object should be evicted on close
*
* \return \herr_t
*
* \details The library's metadata cache is fairly conservative about holding
* on to HDF5 object metadata(object headers, chunk index structures,
* etc.), which can cause the cache size to grow, resulting in memory
* pressure on an application or system. When enabled, the "evict on
* close" property will cause all metadata for an object to be evicted
* from the cache as long as metadata is not referenced by any other
* open object.
*
* This function only applies to file access property lists.
*
* The default library behavior is to not evict on object or file
* close.
*
* When applied to a file access property list, any subsequently opened
* object will inherit the "evict on close" property and will have
* its metadata evicted when the object is closed.
*
* \since 1.10.1
*
*/
H5_DLL herr_t H5Pset_evict_on_close(hid_t fapl_id, hbool_t evict_on_close);
/**
* \ingroup FAPL
*
* \brief Sets offset property for low-level access to a file in a family of
* files
*
* \fapl_id
* \param[in] offset Offset in bytes within the HDF5 file
*
* \return \herr_t
*
* \details H5Pset_family_offset() sets the offset property in the file access
* property list \p fapl_id so that the user application can
* retrieve a file handle for low-level access to a particular member
* of a family of files. The file handle is retrieved with a separate
* call to H5Fget_vfd_handle() (or, in special circumstances, to
* H5FDget_vfd_handle(); see \ref VFL).
*
* The value of \p offset is an offset in bytes from the beginning of
* the HDF5 file, identifying a user-determined location within the
* HDF5 file.
* The file handle the user application is seeking is for the specific
* member-file in the associated family of files to which this offset
* is mapped.
*
* Use of this function is only appropriate for an HDF5 file written as
* a family of files with the \c FAMILY file driver.
*
* \since 1.6.0
*
*/
H5_DLL herr_t H5Pset_family_offset(hid_t fapl_id, hsize_t offset);
/**
* \ingroup FAPL
@@ -3776,9 +4451,279 @@ H5_DLL herr_t H5Pset_family_offset(hid_t fapl_id, hsize_t offset);
*
*/
H5_DLL herr_t H5Pset_fclose_degree(hid_t fapl_id, H5F_close_degree_t degree);
/**
* \ingroup FAPL
*
* \brief Sets an initial file image in a memory buffer
*
* \fapl_id
* \param[in] buf_ptr Pointer to the initial file image, or
* NULL if no initial file image is desired
* \param[in] buf_len Size of the supplied buffer, or
* 0 (zero) if no initial image is desired
*
* \return \herr_t
*
* \details H5Pset_file_image() allows an application to provide a file image
* to be used as the initial contents of a file.
* Calling H5Pset_file_image()makes a copy of the buffer specified in
* \p buf_ptr of size \p buf_len.
*
* \par Motivation:
* H5Pset_file_image() and other elements of HDF5 are
* used to load an image of an HDF5 file into system memory and open
* that image as a regular HDF5 file. An application can then use the
* file without the overhead of disk I/O.
*
* \par Recommended Reading:
* This function is part of the file image
* operations feature set. It is highly recommended to study the guide
* [<em>HDF5 File Image Operations</em>]
* (https://portal.hdfgroup.org/display/HDF5/HDF5+File+Image+Operations
* ) before using this feature set. See the “See Also” section below
* for links to other elements of HDF5 file image operations.
*
* \see
* \li H5LTopen_file_image()
* \li H5Fget_file_image()
* \li H5Pget_file_image()
* \li H5Pset_file_image_callbacks()
* \li H5Pget_file_image_callbacks()
*
* \li [HDF5 File Image Operations]
* (https://portal.hdfgroup.org/display/HDF5/HDF5+File+Image+Operations)
* in [Advanced Topics in HDF5]
* (https://portal.hdfgroup.org/display/HDF5/Advanced+Topics+in+HDF5)
*
* \li Within H5Pset_file_image_callbacks():
* \li Callback #H5FD_file_image_callbacks_t
* \li Callback #H5FD_file_image_op_t
*
* \version 1.8.13 Fortran subroutine added in this release.
* \since 1.8.9
*
*/
H5_DLL herr_t H5Pset_file_image(hid_t fapl_id, void *buf_ptr, size_t buf_len);
/**
* \ingroup FAPL
*
* \brief Sets the callbacks for working with file images
*
* \note **Motivation:** H5Pset_file_image_callbacks() and other elements
* of HDF5 are used to load an image of an HDF5 file into system
* memory and open that image as a regular HDF5 file. An application
* can then use the file without the overhead of disk I/O.\n
* **Recommended Reading:** This function is part of the file
* image operations feature set. It is highly recommended to study
* the guide [HDF5 File Image Operations]
* (https://portal.hdfgroup.org/display/HDF5/HDF5+File+Image+Operations
* ) before using this feature set. See the “See Also” section below
* for links to other elements of HDF5 file image operations.
*
* \fapl_id
* \param[in,out] callbacks_ptr Pointer to the instance of the
* #H5FD_file_image_callbacks_t structure
*
* \return \herr_t \n
* **Failure Modes**: Due to interactions between this function and
* H5Pset_file_image() and H5Pget_file_image(),
* H5Pset_file_image_callbacks() will fail if a file image has
* already been set in the target file access property list, \p fapl_id.
*
* \details H5Pset_file_image_callbacks() sets callback functions for working
* with file images in memory.
*
* H5Pset_file_image_callbacks() allows an application to control the
* management of file image buffers through user defined callbacks.
* These callbacks can be used in the management of file image buffers
* in property lists and with certain file drivers.
*
* H5Pset_file_image_callbacks() must be used before any file image has
* been set in the file access property list. Once a file image has
* been set, the function will fail.
*
* The callback routines set up by H5Pset_file_image_callbacks() are
* invoked when a new file image buffer is allocated, when an existing
* file image buffer is copied or resized, or when a file image buffer
* is released from use.
*
* Some file drivers allow the use of user-defined callback functions
* for allocating, freeing, and copying the driver’s internal buffer,
* potentially allowing optimizations such as avoiding large \c malloc
* and \c memcpy operations, or to perform detailed logging.
*
* From the perspective of the HDF5 library, the operations of the
* \ref H5FD_file_image_callbacks_t.image_malloc "image_malloc",
* \ref H5FD_file_image_callbacks_t.image_memcpy "image_memcpy",
* \ref H5FD_file_image_callbacks_t.image_realloc "image_realloc", and
* \ref H5FD_file_image_callbacks_t.image_free "image_free" callbacks
* must be identical to those of the
* corresponding C standard library calls (\c malloc, \c memcpy,
* \c realloc, and \c free). While the operations must be identical,
* the file image callbacks have more parameters. The return values
* of \ref H5FD_file_image_callbacks_t.image_malloc "image_malloc" and
* \ref H5FD_file_image_callbacks_t.image_realloc "image_realloc" are identical to
* the return values of \c malloc and \c realloc. The return values of
* \ref H5FD_file_image_callbacks_t.image_malloc "image_malloc" and
* \ref H5FD_file_image_callbacks_t.image_free "image_free" differ from the return
* values of \c memcpy and \c free in that the return values of
* \ref H5FD_file_image_callbacks_t.image_memcpy "image_memcpy" and
* \ref H5FD_file_image_callbacks_t.image_free "image_free" can also indicate failure.
*
* The callbacks and their parameters, along with a struct and
* an \c ENUM required for their use, are described below.
*
* <b>Callback struct and \c ENUM:</b>
*
* The callback functions set up by H5Pset_file_image_callbacks() use
* a struct and an \c ENUM that are defined as follows
*
* The struct #H5FD_file_image_callbacks_t serves as a container
* for the callback functions and a pointer to user-supplied data.
* The struct is defined as follows:
* \snippet H5FDpublic.h H5FD_file_image_callbacks_t_snip
*
* Elements of the #H5FD_file_image_op_t are used by the
* callbacks to invoke certain operations on file images. The ENUM is
* defined as follows:
* \snippet H5FDpublic.h H5FD_file_image_op_t_snip
*
* The elements of the #H5FD_file_image_op_t are used in the following
* callbacks:
*
* - The \ref H5FD_file_image_callbacks_t.image_malloc "image_malloc" callback
* contains a pointer to a function that must appear to HDF5 to have
* functionality identical to that of the standard C library \c malloc() call.
*
* - Signature in #H5FD_file_image_callbacks_t:
* \snippet H5FDpublic.h image_malloc_snip
* \n
* - The \ref H5FD_file_image_callbacks_t.image_memcpy "image_memcpy"
* callback contains a pointer to a function
* that must appear to HDF5 to have functionality identical to that
* of the standard C library \c memcopy() call, except that it returns
* a \p NULL on failure. (The \c memcpy C Library routine is defined
* to return the \p dest parameter in all cases.)
*
* - Setting \ref H5FD_file_image_callbacks_t.image_memcpy "image_memcpy"
* to \c NULL indicates that HDF5 should invoke
* the standard C library \c memcpy() routine when copying buffers.
*
* - Signature in #H5FD_file_image_callbacks_t:
* \snippet H5FDpublic.h image_memcpy_snip
* \n
* - The \ref H5FD_file_image_callbacks_t.image_realloc "image_realloc" callback
* contains a pointer to a function that must appear to HDF5 to have
* functionality identical to that of the standard C library \c realloc() call.
*
* - Setting \ref H5FD_file_image_callbacks_t.image_realloc "image_realloc"
* to \p NULL indicates that HDF5 should
* invoke the standard C library \c realloc() routine when resizing
* file image buffers.
*
* - Signature in #H5FD_file_image_callbacks_t:
* \snippet H5FDpublic.h image_realloc_snip
* \n
* - The \ref H5FD_file_image_callbacks_t.image_free "image_free" callback contains
* a pointer to a function that must appear to HDF5 to have functionality
* identical to that of the standard C library \c free() call, except
* that it will return \c 0 (\c SUCCEED) on success and \c -1 (\c FAIL) on failure.
*
* - Setting \ref H5FD_file_image_callbacks_t.image_free "image_free"
* to \c NULL indicates that HDF5 should invoke
* the standard C library \c free() routine when releasing file image
* buffers.
*
* - Signature in #H5FD_file_image_callbacks_t:
* \snippet H5FDpublic.h image_free_snip
* \n
* - The \ref H5FD_file_image_callbacks_t.udata_copy "udata_copy"
* callback contains a pointer to a function
* that, from the perspective of HDF5, allocates a buffer of suitable
* size, copies the contents of the supplied \p udata into the new
* buffer, and returns the address of the new buffer. The function
* returns NULL on failure. This function is necessary if a non-NULL
* \p udata parameter is supplied, so that property lists containing
* the image callbacks can be copied. If the \p udata parameter below
* is \c NULL, then this parameter should be \c NULL as well.
*
* - Signature in #H5FD_file_image_callbacks_t:
* \snippet H5FDpublic.h udata_copy_snip
* \n
* - The \ref H5FD_file_image_callbacks_t.udata_free "udata_free"
* callback contains a pointer to a function
* that, from the perspective of HDF5, frees a user data block. This
* function is necessary if a non-NULL udata parameter is supplied so
* that property lists containing image callbacks can be discarded
* without a memory leak. If the udata parameter below is \c NULL,
* this parameter should be \c NULL as well.
*
* - Signature in #H5FD_file_image_callbacks_t:
* \snippet H5FDpublic.h udata_free_snip
*
* - \p **udata**, the final field in the #H5FD_file_image_callbacks_t
* struct, provides a pointer to user-defined data. This pointer will
* be passed to the
* \ref H5FD_file_image_callbacks_t.image_malloc "image_malloc",
* \ref H5FD_file_image_callbacks_t.image_memcpy "image_memcpy",
* \ref H5FD_file_image_callbacks_t.image_realloc "image_realloc", and
* \ref H5FD_file_image_callbacks_t.image_free "image_free" callbacks.
* Define udata as \c NULL if no user-defined data is provided.
*
* \since 1.8.9
*
*/
H5_DLL herr_t H5Pset_file_image_callbacks(hid_t fapl_id, H5FD_file_image_callbacks_t *callbacks_ptr);
/**
* \ingroup FAPL
*
* \brief Sets the file locking property values
*
* \fapl_id
* \param[in] use_file_locking Toggle to specify file locking (or not)
* \param[in] ignore_when_disabled Toggle to ignore when disabled (or not)
*
* \return \herr_t
*
* \details H5Pset_file_locking() overrides the default file locking flag
* setting that was set when the library was configured.
*
* This setting can be overridden by the \c HDF5_USE_FILE_LOCKING
* environment variable.
*
* File locking is used when creating/opening a file to prevent
* problematic file accesses.
*
* \since 1.10.7
*
*/
H5_DLL herr_t H5Pset_file_locking(hid_t fapl_id, hbool_t use_file_locking, hbool_t ignore_when_disabled);
/**
* \ingroup FAPL
*
* \brief Sets garbage collecting references flag
*
* \fapl_id
* \param[in] gc_ref Flag setting reference garbage collection to on (1) or off (0)
*
* \return \herr_t
*
* \details H5Pset_gc_references() sets the flag for garbage collecting
* references for the file.
*
* Dataset region references and other reference types use space in an
* HDF5 file's global heap. If garbage collection is on and the user
* passes in an uninitialized value in a reference structure, the heap
* might get corrupted. When garbage collection is off, however, and
* the user re-uses a reference, the previous heap block will be
* orphaned and not returned to the free heap space.
*
* When garbage collection is on, the user must initialize the
* reference structures to 0 or risk heap corruption.
*
* The default value for garbage collecting references is off.
*
*/
H5_DLL herr_t H5Pset_gc_references(hid_t fapl_id, unsigned gc_ref);
/**
* \ingroup FAPL
@@ -3861,10 +4806,11 @@ H5_DLL herr_t H5Pset_gc_references(hid_t fapl_id, unsigned gc_ref);
* <td>\p low=#H5F_LIBVER_V18<br />
* \p high=#H5F_LIBVER_V18</td>
* <td>
* \li The library will create objects with the latest format versions
* available to library release 1.8.x.
* \li The library will create objects with the latest format
* versions available to library release 1.8.x.
* \li API calls that create objects or features that are available
* to versions of the library greater than 1.8.x release will fail.
* to versions of the library greater than 1.8.x release will
* fail.
* \li Earlier versions of the library may not be able to access
* objects created with this setting.</td>
* </tr>
@@ -3872,14 +4818,15 @@ H5_DLL herr_t H5Pset_gc_references(hid_t fapl_id, unsigned gc_ref);
* <td>\p low=#H5F_LIBVER_V18<br />
* \p high=#H5F_LIBVER_V110</td>
* <td>
* \li The library will create objects with the latest format versions
* available to library release 1.8.x.
* \li The library will create objects with the latest format
* versions available to library release 1.8.x.
* \li The library will allow objects to be created with the latest
* format versions available to library release 1.10.x.
* Since 1.10.x is also #H5F_LIBVER_LATEST, there is no upper limit
* on the format versions to use. For example, if a newer format
* version is required to support a feature e.g. virtual dataset,
* this setting will allow the object to be created.
* Since 1.10.x is also #H5F_LIBVER_LATEST, there is no upper
* limit on the format versions to use. For example, if a
* newer format version is required to support a feature e.g.
* virtual dataset, this setting will allow the object to be
* created.
* \li Earlier versions of the library may not be able to access
* objects created with this setting.</td>
* </tr>
@@ -3888,39 +4835,356 @@ H5_DLL herr_t H5Pset_gc_references(hid_t fapl_id, unsigned gc_ref);
* \p high=#H5F_LIBVER_V110
* </td>
* <td>
* \li The library will create objects with the latest format versions
* available to library release 1.10.x.
* \li The library will create objects with the latest format
* versions available to library release 1.10.x.
* \li The library will allow objects to be created with the latest
* format versions available to library release 1.10.x.
* Since 1.10.x is also #H5F_LIBVER_LATEST, there is no upper limit
* on the format versions to use. For example, if a newer format
* version is required to support a feature e.g. virtual dataset,
* this setting will allow the object to be created.
* Since 1.10.x is also #H5F_LIBVER_LATEST, there is no upper
* limit on the format versions to use. For example, if a
* newer format version is required to support a feature e.g.
* virtual dataset, this setting will allow the object to be
* created.
* \li This setting allows users to take advantage of the latest
* features and performance enhancements in the library. However,
* objects written with this setting may be accessible to a smaller
* range of library versions than would be the case if low is set
* to #H5F_LIBVER_EARLIEST.
* \li Earlier versions of the library may not be able to access objects created with this
* features and performance enhancements in the library.
* However, objects written with this setting may be
* accessible to a smaller range of library versions than
* would be the case if low is set to #H5F_LIBVER_EARLIEST.
* \li Earlier versions of the library may not be able to access
* objects created with this
* setting.
* </td>
* </tr>
* </table>
*
* \version 1.10.2 #H5F_LIBVER_V18 added to the enumerated defines in #H5F_libver_t.
* \version 1.10.2 #H5F_LIBVER_V18 added to the enumerated defines in
* #H5F_libver_t.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pset_libver_bounds(hid_t plist_id, H5F_libver_t low, H5F_libver_t high);
/**
* \ingroup FAPL
*
* \brief Set the initial metadata cache configuration in the indicated File
* Access Property List to the supplied value
*
* \fapl_id{plist_id}
* \param[in] config_ptr Pointer to the instance of \p H5AC_cache_config_t
* containing the desired configuration
* \return \herr_t
*
* \details The fields of the #H5AC_cache_config_t structure are shown
* below:
* \snippet H5ACpublic.h H5AC_cache_config_t_snip
* \click4more
*
* \details H5Pset_mdc_config() attempts to set the initial metadata cache
* configuration to the supplied value. It will fail if an invalid
* configuration is detected. This configuration is used when the file
* is opened.
*
* See the overview of the metadata cache in the special topics section
* of the user manual for details on what is being configured. If you
* have not read and understood that documentation, you really should
* not be using this API call.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pset_mdc_config(hid_t plist_id, H5AC_cache_config_t *config_ptr);
/**
* \ingroup FAPL
*
* \brief Sets metadata cache logging options
*
* \fapl_id{plist_id}
* \param[in] is_enabled Whether logging is enabled
* \param[in] location Location of log in UTF-8/ASCII (file path/name)
* (On Windows, this must be ASCII)
* \param[in] start_on_access Whether the logging will begin as soon as the
* file is opened or created
*
* \return \herr_t
*
* \details The metadata cache is a central part of the HDF5 library through
* which all file metadata reads and writes take place. File metadata
* is normally invisible to the user and is used by the library for
* purposes such as locating and indexing data. File metadata should
* not be confused with user metadata, which consists of attributes
* created by users and attached to HDF5 objects such as datasets via
* H5A API calls.
*
* Due to the complexity of the cache, a trace/logging feature has
* been created that can be used by HDF5 developers for debugging and
* performance analysis. The functions that control this functionality
* will normally be of use to a very limited number of developers
* outside of The HDF Group. The functions have been documented to
* help users create logs that can be sent with bug reports.
*
* Control of the log functionality is straightforward. Logging is
* enabled via the H5Pset_mdc_log_options() function,
* which will modify the file access property list used to open or
* create a file. This function has a flag that determines whether
* logging begins at file open or starts in a paused state. Log
* messages can then be controlled via the H5Fstart_mdc_logging()
* and H5Fstop_mdc_logging() function.
*
* H5Pget_mdc_log_options() can be used to examine a file access
* property list, and H5Fget_mdc_logging_status() will return the
* current state of the logging flags.
*
* The log format is described in [<em>Metadata Cache Logging</em>]
* (https://portal.hdfgroup.org/display/HDF5/Fine-tuning+the+Metadata+Cache).
*
* \since 1.10.0
*
*/
H5_DLL herr_t H5Pset_mdc_log_options(hid_t plist_id, hbool_t is_enabled, const char *location,
hbool_t start_on_access);
/**
* \ingroup FAPL
*
* \brief Sets the minimum metadata block size
*
* \fapl_id{fapl_id}
* \param[in] size Minimum size, in bytes, of metadata block allocations
*
* \return \herr_t
*
* \details H5Pset_meta_block_size() sets the minimum size, in bytes, of
* metadata block allocations when #H5FD_FEAT_AGGREGATE_METADATA is set by a VFL
* driver.
* Each raw metadata block is initially allocated to be of the given size.
* Specific metadata objects (e.g., object headers, local heaps, B-trees) are then
* sub-allocated from this block.
*
* The default setting is 2048 bytes, meaning that the library will
* attempt to aggregate metadata in at least 2K blocks in the file.
* Setting the value to zero (\Code{0}) with this function will turn
* off metadata aggregation, even if the VFL driver attempts to use the
* metadata aggregation strategy.
*
* Metadata aggregation reduces the number of small data objects in the file that
* would otherwise be required for metadata. The aggregated block of metadata is
* usually written in a single write action and always in a contiguous block,
* potentially significantly improving library and application performance.
*
* \since 1.4.0
*/
H5_DLL herr_t H5Pset_meta_block_size(hid_t fapl_id, hsize_t size);
/**
* \ingroup FAPL
*
* \brief Sets the number of read attempts in a file access property list
*
* \fapl_id{plist_id}
* \param[in] attempts The number of read attempts. Must be a value greater than \Code{0}
*
* \return \herr_t
*
* \return Failure Modes:
* - When the user sets the number of read attempts to \Code{0}.
* - When the input property list is not a file access property list.
* - When the library is unable to set the number of read attempts in the file access property list.
*
* \details H5Pset_metadata_read_attempts() sets the number of reads that the
* library will try when reading checksummed metadata in an HDF5 file opened
* with SWMR access. When reading such metadata, the library will compare the
* checksum computed for the metadata just read with the checksum stored within
* the piece of checksum. When performing SWMR operations on a file, the
* checksum check might fail when the library reads data on a system that is not
* atomic. To remedy such situations, the library will repeatedly read the piece
* of metadata until the check passes or finally fails the read when the allowed
* number of attempts is reached.
*
* The number of read attempts used by the library will depend on how the file is
* opened and whether the user sets the number of read attempts via this routine:
* - For a file opened with SWMR access:
* - If the user sets the number of attempts to \Code{N}, the library will use \Code{N}.
* - If the user does not set the number of attempts, the library will use the
* default for SWMR access (\Code{100}).
* - For a file opened with non-SWMR access, the library will always use the default
* for non-SWMR access (\Code{1}). The value set via this routine does not have any effect
* during non-SWMR access.
*
* \b Example: The first example illustrates the case in setting the number of read attempts for a file
* opened with SWMR access.
*
* \snippet H5Pset_metadata_read_attempts.c SWMR Access
*
* \b Example: The second example illustrates the case in setting the number of
* read attempts for a file opened with non-SWMR access. The value
* set in the file access property list does not have any effect.
*
* \snippet H5Pset_metadata_read_attempts.c non-SWMR Access
*
* \note \b Motivation: On a system that is not atomic, the library might
* possibly read inconsistent metadata with checksum when performing
* single-writer/multiple-reader (SWMR) operations for an HDF5 file. Upon
* encountering such situations, the library will try reading the metadata
* again to obtain consistent data. This routine provides the means to set
* the number of read attempts other than the library default.
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pset_metadata_read_attempts(hid_t plist_id, unsigned attempts);
/**
* \ingroup FAPL
*
* \brief Specifies type of data to be accessed via the \Code{MULTI} driver,
* enabling more direct access
*
* \fapl_id{fapl_id}
* \param[in] type Type of data to be accessed
*
* \return \herr_t
*
* \details H5Pset_multi_type() sets the \Emph{type of data} property in the file
* access property list \p fapl_id. This setting enables a user
* application to specify the type of data the application wishes to
* access so that the application can retrieve a file handle for
* low-level access to the particular member of a set of \Code{MULTI}
* files in which that type of data is stored. The file handle is
* retrieved with a separate call to H5Fget_vfd_handle() (or, in special
* circumstances, to H5FDget_vfd_handle(); see \ref VFL.
*
* The type of data specified in \p type may be one of the following:
*
* <table>
* <tr>
* <td>#H5FD_MEM_SUPER</td> <td>Super block data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_BTREE</td> <td>B-tree data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_DRAW</td> <td>Dataset raw data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_GHEAP</td> <td>Global heap data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_LHEAP</td> <td>Local Heap data</td>
* </tr>
* <tr>
* <td>#H5FD_MEM_OHDR</td> <td>Object header data</td>
* </tr>
* </table>
*
* This function is for use only when accessing an HDF5 file written as a set of
* files with the \Code{MULTI} file driver.
*
* \since 1.6.0
*/
H5_DLL herr_t H5Pset_multi_type(hid_t fapl_id, H5FD_mem_t type);
/**
* \ingroup FAPL
*
* \brief Sets a callback function to invoke when an object flush occurs in the file
*
* \fapl_id{plist_id}
* \op{func}
* \op_data_in{udata}
*
* \return \herr_t
*
* \details H5Pset_object_flush_cb() sets the callback function to invoke in the
* file access property list \p plist_id whenever an object flush occurs in
* the file. Library objects are group, dataset, and committed
* datatype.
*
* The callback function \p func must conform to the prototype defined below:
* \code
* typedef herr_t (*H5F_flush_cb_t)(hid_t object_id, void *user_data)
* \endcode
*
* The parameters of the callback function, per the above prototypes, are defined as follows:
* - \Code{object_id} is the identifier of the object which has just been flushed.
* - \Code{user_data} is the user-defined input data for the callback function.
*
* \b Example: The example below illustrates the usage of this routine to set
* the callback function to invoke when an object flush occurs.
*
* \include H5Pset_object_flush_cb.c
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pset_object_flush_cb(hid_t plist_id, H5F_flush_cb_t func, void *udata);
/**
* \ingroup FAPL
*
* \brief Sets the maximum size of the data sieve buffer
*
* \fapl_id{fapl_id}
* \param[in] size Maximum size, in bytes, of data sieve buffer
*
* \return \herr_t
*
* \details H5Pset_sieve_buf_size() sets \p size, the maximum size in bytes of the
* data sieve buffer, which is used by file drivers that are capable of
* using data sieving.
*
* The data sieve buffer is used when performing I/O on datasets in the
* file. Using a buffer which is large enough to hold several pieces of
* the dataset being read in for hyperslab selections boosts
* performance by quite a bit.
*
* The default value is set to 64KB, indicating that file I/O for raw
* data reads and writes will occur in at least 64KB blocks. Setting
* the value to zero (\Code{0}) with this API function will turn off
* the data sieving, even if the VFL driver attempts to use that
* strategy.
*
* Internally, the library checks the storage sizes of the datasets in
* the file. It picks the smaller one between the size from the file
* access property and the size of the dataset to allocate the sieve
* buffer for the dataset in order to save memory usage.
*
* \version 1.6.0 The \p size parameter has changed from type \Code{hsize_t} to \Code{size_t}.
*
* \since 1.4.0
*/
H5_DLL herr_t H5Pset_sieve_buf_size(hid_t fapl_id, size_t size);
/**
* \ingroup FAPL
*
* \brief Sets the size of a contiguous block reserved for small data
*
* \fapl_id{fapl_id}
* \param[in] size Maximum size, in bytes, of the small data block.
The default size is \Code{2048}.
*
* \return \herr_t
*
* \details H5Pset_small_data_block_size() reserves blocks of \p size bytes for the
* contiguous storage of the raw data portion of \Emph{small} datasets. The
* HDF5 library then writes the raw data from small datasets to this
* reserved space, thus reducing unnecessary discontinuities within
* blocks of meta data and improving I/O performance.
*
* A small data block is actually allocated the first time a qualifying
* small dataset is written to the file. Space for the raw data portion
* of this small dataset is suballocated within the small data block.
* The raw data from each subsequent small dataset is also written to
* the small data block until it is filled; additional small data
* blocks are allocated as required.
*
* The HDF5 library employs an algorithm that determines whether I/O
* performance is likely to benefit from the use of this mechanism with
* each dataset as storage space is allocated in the file. A larger
* \p size will result in this mechanism being employed with larger
* datasets.
*
* The small data block size is set as an allocation property in the
* file access property list identified by \p fapl_id.
*
* Setting \p size to zero (\Code{0}) disables the small data block mechanism.
*
* \since 1.4.4
*/
H5_DLL herr_t H5Pset_small_data_block_size(hid_t fapl_id, hsize_t size);
/**
* \ingroup FAPL
@@ -3943,14 +5207,242 @@ H5_DLL herr_t H5Pset_small_data_block_size(hid_t fapl_id, hsize_t size);
H5_DLL herr_t H5Pset_vol(hid_t plist_id, hid_t new_vol_id, const void *new_vol_info);
#ifdef H5_HAVE_PARALLEL
/**
* \ingroup GACPL
*
* \brief Sets metadata I/O mode for read operations to collective or independent (default)
*
* \gacpl_id
* \param[in] is_collective Boolean value indicating whether metadata reads are collective
* (\Code{1}) or independent (\Code{0}).
* Default mode: Independent (\Code{0})
*
* \return \herr_t
*
* \details H5Pset_all_coll_metadata_ops() sets the metadata I/O mode for read
* operations in the access property list \p plist_id.
*
* When engaging in parallel I/O, all metadata write operations must be
* collective. If \p is_collective is \Code{1}, this property specifies
* that the HDF5 library will perform all metadata read operations
* collectively; if \p is_collective is \Code{0}, such operations may
* be performed independently.
*
* Users must be aware that several HDF5 operations can potentially
* issue metadata reads. These include opening a dataset, datatype, or
* group; reading an attribute; or issuing a \Emph{get info} call such
* as getting information for a group with H5Fget_info(). Collective
* I/O requirements must be kept in mind when issuing such calls in the
* context of parallel I/O.
*
* If this property is collective on a file access property list that
* is used in creating or opening a file, then the HDF5 library will
* assume that all metadata read operations issued on that file
* identifier will be issued collectively from all ranks irrespective
* of the individual setting of a particular operation. If this
* assumption is not adhered to, corruption will be introduced in the
* metadata cache and HDF5’s behavior will be undefined.
*
* Alternatively, a user may wish to avoid setting this property
* globally on the file access property list, and individually set it
* on particular object access property lists (dataset, group, link,
* datatype, attribute access property lists) for certain operations.
* This will indicate that only the operations issued with such an
* access property list will be called collectively and other
* operations may potentially be called independently. There are,
* however, several HDF5 operations that can issue metadata reads but
* have no property list in their function signatures to allow passing
* the collective requirement property. For those operations, the only
* option is to set the global collective requirement property on the
* file access property list; otherwise the metadata reads that can be
* triggered from those operations will be done independently by each
* process.
*
* Functions that do not accommodate an access property list but that
* might issue metadata reads are listed in \ref maybe_metadata_reads.
*
* \attention As noted above, corruption will be introduced into the metadata
* cache and HDF5 library behavior will be undefined when both of the following
* conditions exist:
* - A file is created or opened with a file access property list in which the
* collective metadata I/O property is set to \Code{1}.
* - Any function is called that triggers an independent metadata read while the
* file remains open with that file access property list.
*
* \attention An approach that avoids this corruption risk is described above.
*
* \sa_metadata_ops
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pset_all_coll_metadata_ops(hid_t plist_id, hbool_t is_collective);
/**
* \ingroup GACPL
*
* \brief Retrieves metadata read mode setting
*
* \gacpl_id
* \param[out] is_collective Pointer to a buffer containing the Boolean value indicating whether metadata
* reads are collective (\Code{>0}) or independent (\Code{0}).
* Default mode: Independent (\Code{0})
*
* \return \herr_t
*
* \details H5Pget_all_coll_metadata_ops() retrieves the collective metadata read setting from the access
* property list \p plist_id into \p is_collective.
*
* \sa_metadata_ops
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pget_all_coll_metadata_ops(hid_t plist_id, hbool_t *is_collective);
/**
* \ingroup FAPL
*
* \brief Sets metadata write mode to collective or independent (default)
*
* \fapl_id{plist_id}
* \param[out] is_collective Boolean value indicating whether metadata
* writes are collective (\Code{>0}) or independent (\Code{0}).
* \Emph{Default mode:} Independent (\Code{0})
* \return \herr_t
*
* \details H5Pset_coll_metadata_write() tells the HDF5 library whether to
* perform metadata writes collectively (1) or independently (0).
*
* If collective access is selected, then on a flush of the metadata
* cache, all processes will divide the metadata cache entries to be
* flushed evenly among themselves and issue a single MPI-IO collective
* write operation. This is the preferred method when the size of the
* metadata created by the application is large.
*
* If independent access is selected, the library uses the default
* method for doing metadata I/O either from process zero or
* independently from each process.
*
* \sa_metadata_ops
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pset_coll_metadata_write(hid_t plist_id, hbool_t is_collective);
/**
* \ingroup FAPL
*
* \brief Retrieves metadata write mode setting
*
* \fapl_id{plist_id}
* \param[out] is_collective Pointer to a boolean value indicating whether
* metadata writes are collective (\Code{>0}) or independent (\Code{0}).
* \Emph{Default mode:} Independent (\Code{0})
* \return \herr_t
*
* \details H5Pget_coll_metadata_write() retrieves the collective metadata write
* setting from the file access property into \p is_collective.
*
* \sa_metadata_ops
*
* \since 1.10.0
*/
H5_DLL herr_t H5Pget_coll_metadata_write(hid_t plist_id, hbool_t *is_collective);
/**
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pget_mpi_params(hid_t fapl_id, MPI_Comm *comm, MPI_Info *info);
/**
* \todo Add missing documentation
*/
H5_DLL herr_t H5Pset_mpi_params(hid_t fapl_id, MPI_Comm comm, MPI_Info info);
#endif /* H5_HAVE_PARALLEL */
/**
* \ingroup FAPL
*
* \brief Sets the metadata cache image option for a file access property list
*
* \fapl_id{plist_id}
* \param[out] config_ptr Pointer to metadata cache image configuration values
* \return \herr_t
*
* \details H5Pset_mdc_image_config() sets the metadata cache image option with
* configuration values specified by \p config_ptr for the file access
* property list specified in \p plist_id.
*
* #H5AC_cache_image_config_t is defined as follows:
* \snippet H5ACpublic.h H5AC_cache_image_config_t_snip
* \click4more
*
* \par Limitations: While it is an obvious error to request a cache image when
* opening the file read only, it is not in general possible to test for
* this error in the H5Pset_mdc_image_config() call. Rather than fail the
* subsequent file open, the library silently ignores the file image
* request in this case.\n It is also an error to request a cache image on
* a file that does not support superblock extension messages (i.e. a
* superblock version less than 2). As above, it is not always possible to
* detect this error in the H5Pset_mdc_image_config() call, and thus the
* request for a cache image will fail silently in this case as well.\n
* Creation of cache images is currently disabled in parallel -- as above,
* any request for a cache image in this context will fail silently.\n
* Files with cache images may be read in parallel applications, but note
* that the load of the cache image is a collective operation triggered by
* the first operation that accesses metadata after file open (or, if
* persistent free space managers are enabled, on the first allocation or
* deallocation of file space, or read of file space manager status,
* whichever comes first). Thus the parallel process may deadlock if any
* process does not participate in this access.\n
* In long sequences of file closes and opens, infrequently accessed
* metadata can accumulate in the cache image to the point where the cost
* of storing and restoring this metadata exceeds the benefit of retaining
* frequently used metadata in the cache image. When implemented, the
* #H5AC_cache_image_config_t::entry_ageout should address this problem. In
* the interim, not requesting a cache image every n file close/open cycles
* may be an acceptable work around. The choice of \c n will be driven by
* application behavior, but \Code{n = 10} seems a good starting point.
*
* \since 1.10.1
*/
H5_DLL herr_t H5Pset_mdc_image_config(hid_t plist_id, H5AC_cache_image_config_t *config_ptr);
/**
* \ingroup FAPL
*
* \brief Sets the maximum size for the page buffer and the minimum percentage
* for metadata and raw data pages
*
* \fapl_id{plist_id}
* \param[in] buf_size Maximum size, in bytes, of the page buffer
* \param[in] min_meta_per Minimum metadata percentage to keep in the page buffer
* before allowing pages containing metadata to be evicted (Default is 0)
* \param[in] min_raw_per Minimum raw data percentage to keep in the page buffer
* before allowing pages containing raw data to be evicted (Default is 0)
* \return \herr_t
*
* \details H5Pset_page_buffer_size() sets buf_size, the maximum size in bytes
* of the page buffer. The default value is zero, meaning that page
* buffering is disabled. When a non-zero page buffer size is set, the
* library will enable page buffering if that size is larger or equal
* than a single page size if a paged file space strategy is enabled
* using the functions H5Pset_file_space_strategy() and
* H5Pset_file_space_page_size().
*
* The page buffer layer captures all I/O requests before they are
* issued to the VFD and "caches" them in fixed sized pages. Once the
* total number of pages exceeds the page buffer size, the library
* evicts pages from the page buffer by writing them to the VFD. At
* file close, the page buffer is flushed writing all the pages to the
* file.
*
* If a non-zero page buffer size is set, and the file space strategy
* is not set to paged or the page size for the file space strategy is
* larger than the page buffer size, the subsequent call to H5Fcreate()
* or H5Fopen() using the \p plist_id will fail.
*
* The function also allows setting the minimum percentage of pages for
* metadata and raw data to prevent a certain type of data to evict hot
* data of the other type.
*
* \since 1.10.1
*
*/
H5_DLL herr_t H5Pset_page_buffer_size(hid_t plist_id, size_t buf_size, unsigned min_meta_per,
unsigned min_raw_per);
@@ -4288,7 +5780,7 @@ H5_DLL H5D_layout_t H5Pget_layout(hid_t plist_id);
* virtual dataset that has the creation property list specified
* by \p dcpl_id.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -4331,7 +5823,7 @@ H5_DLL herr_t H5Pget_virtual_count(hid_t dcpl_id, size_t *count /*out*/);
* assigned to \p size for a second H5Pget_virtual_dsetname()
* call, which will retrieve the actual dataset name.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -4375,7 +5867,7 @@ H5_DLL ssize_t H5Pget_virtual_dsetname(hid_t dcpl_id, size_t index, char *name /
* \p size for a second H5Pget_virtual_filename() call, which
* will retrieve the actual filename.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -4400,7 +5892,7 @@ H5_DLL ssize_t H5Pget_virtual_filename(hid_t dcpl_id, size_t index, char *name /
* index, \p index, and returns a dataspace identifier for the
* selection within the source dataset used in the mapping.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -4425,7 +5917,7 @@ H5_DLL hid_t H5Pget_virtual_srcspace(hid_t dcpl_id, size_t index);
* index, \p index, and returns a dataspace identifier for the
* selection within the virtual dataset used in the mapping.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -5159,6 +6651,133 @@ H5_DLL herr_t H5Pset_scaleoffset(hid_t plist_id, H5Z_SO_scale_type_t scale_type,
*
*/
H5_DLL herr_t H5Pset_szip(hid_t plist_id, unsigned options_mask, unsigned pixels_per_block);
/**
* \ingroup DCPL
*
* \brief Sets the mapping between virtual and source datasets
*
* \dcpl_id
* \param[in] vspace_id The dataspace identifier with the selection within the
* virtual dataset applied, possibly an unlimited selection
* \param[in] src_file_name The name of the HDF5 file where the source dataset is
* located or a \Code{"."} (period) for a source dataset in the same
* file. The file might not exist yet. The name can be specified using
* a C-style \c printf statement as described below.
* \param[in] src_dset_name The path to the HDF5 dataset in the file specified by
* \p src_file_name. The dataset might not exist yet. The dataset name
* can be specified using a C-style \c printf statement as described below.
* \param[in] src_space_id The source dataset’s dataspace identifier with a
* selection applied, possibly an unlimited selection
* \return \herr_t
*
* \details H5Pset_virtual() maps elements of the virtual dataset (VDS)
* described by the virtual dataspace identifier \p vspace_id to the
* elements of the source dataset described by the source dataset
* dataspace identifier \p src_space_id. The source dataset is
* identified by the name of the file where it is located,
* \p src_file_name, and the name of the dataset, \p src_dset_name.
*
* \par C-style \c printf Formatting Statements:
* C-style \c printf formatting allows a pattern to be specified in the name
* of a source file or dataset. Strings for the file and dataset names are
* treated as literals except for the following substitutions:
* <table>
* <tr>
* <td>\Code{"%%"}</td>
* <td>Replaced with a single \Code{"%"} (percent) character.</td>
* </tr>
* <tr>
* <td><code>"%<d>b"</code></td>
* <td>Where <code>"<d>"</code> is the virtual dataset dimension axis (0-based)
* and \Code{"b"} indicates that the block count of the selection in that
* dimension should be used. The full expression (for example, \Code{"%0b"})
* is replaced with a single numeric value when the mapping is evaluated at
* VDS access time. Example code for many source and virtual dataset mappings
* is available in the "Examples of Source to Virtual Dataset Mapping"
* chapter in the
* <a href="https://portal.hdfgroup.org/display/HDF5/RFC+HDF5+Virtual+Dataset">
* RFC: HDF5 Virtual Dataset</a>.
* </td>
* </tr>
* </table>
* If the printf form is used for the source file or dataset names, the
* selection in the source dataset’s dataspace must be fixed-size.
*
* \par Source File Resolutions:
* When a source dataset residing in a different file is accessed, the
* library will search for the source file \p src_file_name as described
* below:
* \li If \p src_file_name is a \Code{"."} (period) then it refers to the
* file containing the virtual dataset.
* \li If \p src_file_name is a relative pathname, the following steps are
* performed:
* - The library will get the prefix(es) set in the environment
* variable \c HDF5_VDS_PREFIX and will try to prepend each prefix
* to \p src_file_name to form a new \p src_file_name. If the new
* \p src_file_name does not exist or if \c HDF5_VDS_PREFIX is not
* set, the library will get the prefix set via H5Pset_virtual_prefix()
* and prepend it to \p src_file_name to form a new \p src_file_name.
* If the new \p src_file_name does not exist or no prefix is being
* set by H5Pset_virtual_prefix() then the path of the file containing
* the virtual dataset is obtained. This path can be the absolute path
* or the current working directory plus the relative path of that
* file when it is created/opened. The library will prepend this path
* to \p src_file_name to form a new \p src_file_name.
* - If the new \p src_file_name does not exist, then the library will
* look for \p src_file_name and will return failure/success accordingly.
* \li If \p src_file_name is an absolute pathname, the library will first
* try to find \p src_file_name. If \p src_file_name does not exist,
* \p src_file_name is stripped of directory paths to form a new
* \p src_file_name. The search for the new \p src_file_name then follows
* the same steps as described above for a relative pathname. See
* examples below illustrating how \p src_file_name is stripped to form
* a new \p src_file_name.
* \par
* Note that \p src_file_name is considered to be an absolute pathname when
* the following condition is true:
* \li For Unix, the first character of \p src_file_name is a slash
* (\Code{/}).\n For example, consider a \p src_file_name of
* \Code{/tmp/A.h5}. If that source file does not exist, the new
* \p src_file_name after stripping will be \Code{A.h5}.
* \li For Windows, there are 6 cases:
* 1. \p src_file_name is an absolute drive with absolute pathname.\n
* For example, consider a \p src_file_name of \Code{/tmp/A.h5}.
* If that source file does not exist, the new \p src_file_name
* after stripping will be \Code{A.h5}.
* 2. \p src_file_name is an absolute pathname without specifying
* drive name.\n For example, consider a \p src_file_name of
* \Code{/tmp/A.h5}. If that source file does not exist, the new
* \p src_file_name after stripping will be \Code{A.h5}.
* 3. \p src_file_name is an absolute drive with relative pathname.\n
* For example, consider a \p src_file_name of \Code{/tmp/A.h5}.
* If that source file does not exist, the new \p src_file_name
* after stripping will be \Code{tmp/A.h5}.
* 4. \p src_file_name is in UNC (Uniform Naming Convention) format
* with server name, share name, and pathname.\n
* For example, consider a \p src_file_name of \Code{/tmp/A.h5}.
* If that source file does not exist, the new \p src_file_name
* after stripping will be \Code{A.h5}.
* 5. \p src_file_name is in Long UNC (Uniform Naming Convention)
* format with server name, share name, and pathname.\n
* For example, consider a \p src_file_name of \Code{/tmp/A.h5}.
* If that source file does not exist, the new \p src_file_name
* after stripping will be \Code{A.h5}.
* 6. \p src_file_name is in Long UNC (Uniform Naming Convention)
* format with an absolute drive and an absolute pathname.\n
* For example, consider a \p src_file_name of \Code{/tmp/A.h5}.
* If that source file does not exist, the new \p src_file_name
* after stripping will be \Code{A.h5}
*
* \see <a href="https://portal.hdfgroup.org/display/HDF5/Virtual+Dataset++-+VDS">
* Virtual Dataset Overview</a>
*
* \see_virtual
*
* \version 1.10.2 A change was made to the method of searching for VDS source files.
* \since 1.10.0
*
*/
H5_DLL herr_t H5Pset_virtual(hid_t dcpl_id, hid_t vspace_id, const char *src_file_name,
const char *src_dset_name, hid_t src_space_id);
@@ -5191,8 +6810,6 @@ H5_DLL herr_t H5Pset_virtual(hid_t dcpl_id, hid_t vspace_id, const char *src_fil
* \p udata is the user-defined input data for the callback
* function.
*
* \todo Example Usage was removed and needs to be re-added
*
* \since 1.10.0
*
*/
@@ -5298,7 +6915,7 @@ H5_DLL ssize_t H5Pget_efile_prefix(hid_t dapl_id, char *prefix /*out*/, size_t s
* NULL will return the size of the prefix without the NULL
* terminator.
*
* \virtual
* \see_virtual
*
* \since 1.10.2
*
@@ -5348,7 +6965,7 @@ H5_DLL herr_t H5Pget_virtual_printf_gap(hid_t dapl_id, hsize_t *gap_size);
* list, \p dapl_id, and retrieves the flag, \p view, set by the
* H5Pset_virtual_view() call.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -5389,7 +7006,7 @@ H5_DLL herr_t H5Pget_virtual_view(hid_t dapl_id, H5D_vds_view_t *view);
* \p boundary. It is a 1-dimensional array with \p ndims
* elements, which should be the same as the rank of the
* dataset’s dataspace. While appending to a dataset along a
* particular dimension index via H5DOappend(), the library
* particular dimension index via H5Dappend(), the library
* determines a boundary is reached when the resulting dimension
* size is divisible by \p boundary[index]. A zero value for
* \p boundary[index] indicates no boundary is set for that
@@ -5415,8 +7032,7 @@ H5_DLL herr_t H5Pget_virtual_view(hid_t dapl_id, H5D_vds_view_t *view);
*
* The callback function \p func must conform to the following
* prototype:
* \Code{typedef herr_t (#H5D_append_cb_t)(hid_t dataset_id,
* hsize_t *cur_dims, void *user_data)}
* \snippet H5Dpublic.h H5D_append_cb_t_snip
*
* The parameters of the callback function, per the above
* prototype, are defined as follows:
@@ -5426,10 +7042,6 @@ H5_DLL herr_t H5Pget_virtual_view(hid_t dapl_id, H5D_vds_view_t *view);
* a boundary is hit.
* \li \p user_data is the user-defined input data.
*
* \todo Example Usage was removed and should be added back.
* \todo Adding snippet for H5D_append_cb_t_snip did not work.
* \todo H5DOappend() not found
*
* \since 1.10.0
*
*/
@@ -5643,7 +7255,7 @@ H5_DLL herr_t H5Pset_efile_prefix(hid_t dapl_id, const char *prefix);
* buffer should not be freed until the property list has been
* closed.
*
* \virtual
* \see_virtual
*
* \since 1.10.2
*
@@ -5685,7 +7297,7 @@ H5_DLL herr_t H5Pset_virtual_prefix(hid_t dapl_id, const char *prefix);
* d-2 and d-3 will be filled according to the virtual
* dataset’s fill value setting.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -5723,7 +7335,7 @@ H5_DLL herr_t H5Pset_virtual_printf_gap(hid_t dapl_id, hsize_t gap_size);
* Missing mapped data is filled with the fill value set in the
* VDS creation property list.
*
* \virtual
* \see_virtual
*
* \since 1.10.0
*
@@ -5731,8 +7343,48 @@ H5_DLL herr_t H5Pset_virtual_printf_gap(hid_t dapl_id, hsize_t gap_size);
H5_DLL herr_t H5Pset_virtual_view(hid_t dapl_id, H5D_vds_view_t view);
/* Dataset xfer property list (DXPL) routines */
/**
*
* \ingroup DXPL
*
* \brief Gets B-tree split ratios for a dataset transfer property list
*
* \dxpl_id{plist_id}
* \param[out] left The B-tree split ratio for left-most nodes
* \param[out] middle The B-tree split ratio for right-most nodes and lone nodes
* \param[out] right The B-tree split ratio for all other nodes
* \return \herr_t
*
* \details H5Pget_btree_ratios() returns the B-tree split ratios for a dataset
* transfer property list.
*
* The B-tree split ratios are returned through the non-NULL arguments
* \p left, \p middle, and \p right, as set by the H5Pset_btree_ratios()
* function.
*
*/
H5_DLL herr_t H5Pget_btree_ratios(hid_t plist_id, double *left /*out*/, double *middle /*out*/,
double *right /*out*/);
/**
*
* \ingroup DXPL
*
* \brief Reads buffer settings
*
* \param[in] plist_id Identifier for the dataset transfer property list
* \param[out] tconv Address of the pointer to application-allocated type
* conversion buffer
* \param[out] bkg Address of the pointer to application-allocated
* background buffer
*
* \return Returns buffer size, in bytes, if successful; otherwise 0 on failure.
*
* \details H5Pget_buffer() reads values previously set with H5Pset_buffer().
*
* \version 1.6.0 The return type changed from \p hsize_t to \p size_t.
* \version 1.4.0 The return type changed to \p hsize_t.
*
*/
H5_DLL size_t H5Pget_buffer(hid_t plist_id, void **tconv /*out*/, void **bkg /*out*/);
/**
*
@@ -5740,54 +7392,237 @@ H5_DLL size_t H5Pget_buffer(hid_t plist_id, void **tconv /*out*/, void **bkg /*o
*
* \brief Retrieves a data transform expression
*
* \param[in] plist_id Identifier of the property list or class
* \param[out] expression Pointer to memory where the transform expression
* will be copied
* \param[in] size Number of bytes of the transform expression to copy to
* \param[in] plist_id Identifier of the property list or class
* \param[out] expression Pointer to memory where the transform expression will
* be copied
* \param[in] size Number of bytes of the transform expression to copy
* to
*
* \return Returns the size of the transform expression if successful;
* otherwise returns a negative value.
* \return Success: the size of the transform expression. Failure: a negative
* value.
*
* \details H5Pget_data_transform() retrieves the data transform
* expression previously set in the dataset transfer property
* list \p plist_id by H5Pset_data_transform().
* \details H5Pget_data_transform() retrieves the data transform expression
* previously set in the dataset transfer property list \p plist_id
* by H5Pset_data_transform().
*
* H5Pget_data_transform() can be used to both retrieve the
* transform expression and to query its size.
* H5Pget_data_transform() can be used to both retrieve the transform
* expression and query its size.
*
* If \p expression is non-NULL, up to \p size bytes of the data
* transform expression are written to the buffer. If
* \p expression is NULL, \p size is ignored and the function
* does not write anything to the buffer. The function always
* returns the size of the data transform expression.
* transform expression are written to the buffer. If \p expression
* is NULL, \p size is ignored, and the function does not write
* anything to the buffer. The function always returns the size of
* the data transform expression.
*
* If 0 is returned for the size of the expression, no data
* transform expression exists for the property list.
* If 0 is returned for the size of the expression, no data transform
* expression exists for the property list.
*
* If an error occurs, the buffer pointed to by \p expression is
* unchanged and the function returns a negative value.
* unchanged, and the function returns a negative value.
*
* \par Example
* An example snippet from examples/h5_dtransform.c:
* \snippet h5_dtransform.c H5Pget_data_transform_snip
*
* \since 1.8.0
*
*/
H5_DLL ssize_t H5Pget_data_transform(hid_t plist_id, char *expression /*out*/, size_t size);
H5_DLL ssize_t H5Pget_data_transform(hid_t plist_id, char *expression /*out*/, size_t size);
/**
*
* \ingroup DXPL
*
* \brief Determines whether error-detection is enabled for dataset reads
*
* \param[in] plist_id Dataset transfer property list identifier
*
* \return Returns \p H5Z_ENABLE_EDC or \p H5Z_DISABLE_EDC if successful;
* otherwise returns a negative value.
*
* \details H5Pget_edc_check() queries the dataset transfer property
* list \p plist to determine whether error detection is enabled for
* data read operations.
*
* \since 1.6.0
*
*/
H5_DLL H5Z_EDC_t H5Pget_edc_check(hid_t plist_id);
H5_DLL herr_t H5Pget_hyper_vector_size(hid_t fapl_id, size_t *size /*out*/);
H5_DLL int H5Pget_preserve(hid_t plist_id);
H5_DLL herr_t H5Pget_type_conv_cb(hid_t dxpl_id, H5T_conv_except_func_t *op, void **operate_data);
H5_DLL herr_t H5Pget_vlen_mem_manager(hid_t plist_id, H5MM_allocate_t *alloc_func, void **alloc_info,
H5MM_free_t *free_func, void **free_info);
H5_DLL herr_t H5Pset_btree_ratios(hid_t plist_id, double left, double middle, double right);
H5_DLL herr_t H5Pset_buffer(hid_t plist_id, size_t size, void *tconv, void *bkg);
/**
*
* \ingroup DXPL
*
* \brief Retrieves number of I/O vectors to be read/written in hyperslab I/O
*
* \param[in] fapl_id Dataset transfer property list identifier
* \param[out] size Number of I/O vectors to accumulate in memory for I/O operations
*
* \return \herr_t
*
* \details H5Pget_hyper_vector_size() retrieves the number of I/O vectors to be accumulated in
* memory before being issued to the lower levels of the HDF5 library for reading or
* writing the actual data.
*
* The number of I/O vectors set in the dataset transfer property list \p fapl_id is
* returned in \p size. Unless the default value is in use, \p size was
* previously set with a call to H5Pset_hyper_vector_size().
*
* \since 1.6.0
*
*/
H5_DLL herr_t H5Pget_hyper_vector_size(hid_t fapl_id, size_t *size /*out*/);
/**
*
* \ingroup DXPL
*
* \brief Checks status of the dataset transfer property list (\b DEPRECATED)
*
* \deprecated{H5Pget_preserve() is deprecated as it is no longer useful;
* compound datatype field preservation is now core functionality
* in the HDF5 library.}
*
* \param[in] plist_id Identifier for the dataset transfer property list
*
* \return Returns 1 or 0 if successful; otherwise returns a negative value.
*
* \details H5Pget_preserve() checks the status of the dataset transfer
* property list.
*
* \version 1.6.0 The flag parameter was changed from INTEGER to LOGICAL to
* better match the C API. (Fortran 90)
*
*/
H5_DLL int H5Pget_preserve(hid_t plist_id);
/**
*
* \ingroup DXPL
*
* \brief Gets user-defined datatype conversion callback function
*
* \param[in] dxpl_id Dataset transfer property list identifier
* \param[out] op User-defined type conversion callback function
* \param[out] operate_data User-defined input data for the callback function
*
* \return \herr_t
*
* \details H5Pget_type_conv_cb() gets the user-defined datatype conversion
* callback function \p op in the dataset transfer property list
* \p dxpl_id.
*
* The parameter \p operate_data is a pointer to user-defined input
* data for the callback function.
*
* The callback function \p op defines the actions an application is
* to take when there is an exception during datatype conversion.
*
* Please refer to the function H5Pset_type_conv_cb() for more details.
*
*/
H5_DLL herr_t H5Pget_type_conv_cb(hid_t dxpl_id, H5T_conv_except_func_t *op, void **operate_data);
/**
*
* \ingroup DXPL
*
* \brief Gets the memory manager for variable-length datatype allocation in H5Dread() and H5Dvlen_reclaim()
*
* \param[in] plist_id Identifier for the dataset transfer property list
* \param[out] alloc_func User's allocate routine, or NULL for system malloc
* \param[out] alloc_info Extra parameter for user’s allocation routine.
* Contents are ignored if preceding
* parameter is NULL \param[out] free_func User's free routine, or NULL for
* system free \param[out] free_info
* Extra parameter for user’s free routine. Contents are ignored if preceding
* parameter is NULL
*
* \return \herr_t
*
* \details H5Pget_vlen_mem_manager() is the companion function to
* H5Pset_vlen_mem_manager(), returning the parameters set by
* that function.
*
*/
H5_DLL herr_t H5Pget_vlen_mem_manager(hid_t plist_id, H5MM_allocate_t *alloc_func, void **alloc_info,
H5MM_free_t *free_func, void **free_info);
/**
*
* \ingroup DXPL
*
* \brief Sets B-tree split ratios for a dataset transfer property list
*
* \param[in] plist_id The dataset transfer property list identifier
* \param[in] left The B-tree split ratio for left-most nodes
* \param[in] middle The B-tree split ratio for all other nodes
* \param[in] right The B-tree split ratio for right-most nodes and lone
* nodes
*
* \return \herr_t
*
* \details H5Pset_btree_ratios() sets the B-tree split ratios for a dataset
* transfer property list. The split ratios determine what percent of
* children go in the first node when a node splits.
*
* The ratio \p left is used when the splitting node is the left-most
* node at its level in the tree;
* the ratio \p right is used when the splitting node is the right-most
* node at its level; and
* the ratio \p middle is used for all other cases.
*
* A node that is the only node at its level in the tree uses the
* ratio \p right when it splits.
*
* All ratios are real numbers between 0 and 1, inclusive.
*
*/
H5_DLL herr_t H5Pset_btree_ratios(hid_t plist_id, double left, double middle, double right);
/**
*
* \ingroup DXPL
*
* \brief Sets type conversion and background buffers
*
* \dxpl_id{plist_id}
* \param[in] size Size, in bytes, of the type conversion and background buffers
* \param[in] tconv Pointer to application-allocated type conversion buffer
* \param[in] bkg Pointer to application-allocated background buffer
* \return \herr_t
*
* \details Given a dataset transfer property list, H5Pset_buffer() sets the
* maximum size for the type conversion buffer and background buffer
* and optionally supplies pointers to application-allocated
* buffers. If the buffer size is smaller than the entire amount of
* data being transferred between the application and the file, and a
* type conversion buffer or background buffer is required, then strip
* mining will be used.
*
* Note that there are minimum size requirements for the buffer. Strip
* mining can only break the data up along the first dimension, so the
* buffer must be large enough to accommodate a complete slice that
* encompasses all of the remaining dimensions. For example, when strip
* mining a \Code{100x200x300} hyperslab of a simple data space, the
* buffer must be large enough to hold \Code{1x200x300} data
* elements. When strip mining a \Code{100x200x300x150} hyperslab of a
* simple data space, the buffer must be large enough to hold
* \Code{1x200x300x150} data elements.
*
* If \p tconv and/or \p bkg are null pointers, then buffers will be
* allocated and freed during the data transfer.
*
* The default value for the maximum buffer is 1 MiB.
*
* \version 1.6.0 The \p size parameter has changed from type hsize_t to \c size_t.
* \version 1.4.0 The \p size parameter has changed to type hsize_t.
*
*/
H5_DLL herr_t H5Pset_buffer(hid_t plist_id, size_t size, void *tconv, void *bkg);
/**
* \ingroup DXPL
*
* \brief Sets a data transform expression
*
* \param[in] plist_id Identifier of the property list or class
* \dxpl_id{plist_id}
* \param[in] expression Pointer to the null-terminated data transform
* expression
*
* \return \herr_t
*
* \details H5Pset_data_transform() sets the data transform to be used for
@@ -5795,11 +7630,11 @@ H5_DLL herr_t H5Pset_buffer(hid_t plist_id, size_t size, void *tconv, void *b
* transfer property list \p plist_id.
*
* The \p expression parameter is a string containing an algebraic
* expression, such as (5/9.0)*(x-32) or x*(x-5). When a dataset
* is read or written with this property list, the transform
* expression is applied with the x being replaced by the values
* in the dataset. When reading data, the values in the file are
* not changed and the transformed data is returned to the user.
* expression, such as \Code{(5/9.0)*(x-32)} or \Code{x*(x-5)}. When a
* dataset is read or written with this property list, the transform
* expression is applied with the \c x being replaced by the values in
* the dataset. When reading data, the values in the file are not
* changed and the transformed data is returned to the user.
*
* Data transforms can only be applied to integer or
* floating-point datasets. Order of operations is obeyed and
@@ -5813,17 +7648,354 @@ H5_DLL herr_t H5Pset_buffer(hid_t plist_id, size_t size, void *tconv, void *b
*
*/
H5_DLL herr_t H5Pset_data_transform(hid_t plist_id, const char *expression);
/**
* \ingroup DXPL
*
* \brief Sets the dataset transfer property list to enable or disable error
* detection when reading data
*
* \dxpl_id{plist_id}
* \param[in] check Specifies whether error checking is enabled or disabled
* for dataset read operations
* \return \herr_t
*
* \details H5Pset_edc_check() sets the dataset transfer property list \p plist
* to enable or disable error detection when reading data.
*
* Whether error detection is enabled or disabled is specified in the
* \p check parameter. Valid values are #H5Z_ENABLE_EDC (default) and
* #H5Z_DISABLE_EDC.
*
* \note The initial error detection implementation, Fletcher32 checksum,
* supports error detection for chunked datasets only.
*
* \attention The Fletcher32 EDC checksum filter, set with H5Pset_fletcher32(),
* was added in HDF5 Release 1.6.0. In the original implementation,
* however, the checksum value was calculated incorrectly on
* little-endian systems. The error was fixed in HDF5 Release 1.6.3.\n
* As a result of this fix, an HDF5 library of Release 1.6.0 through
* Release 1.6.2 cannot read a dataset created or written with
* Release 1.6.3 or later if the dataset was created with the
* checksum filter and the filter is enabled in the reading
* library. (Libraries of Release 1.6.3 and later understand the
* earlier error and compensate appropriately.)\n
* \Bold{Work-around:} An HDF5 library of Release 1.6.2 or earlier
* will be able to read a dataset created or written with the
* checksum filter by an HDF5 library of Release 1.6.3 or later if
* the checksum filter is disabled for the read operation. This can
* be accomplished via an H5Pset_edc_check() call with the value
* #H5Z_DISABLE_EDC in the second parameter. This has the obvious
* drawback that the application will be unable to verify the
* checksum, but the data does remain accessible.
*
* \version 1.6.3 Error in checksum calculation on little-endian systems
* corrected in this release.
* \since 1.6.0
*
*/
H5_DLL herr_t H5Pset_edc_check(hid_t plist_id, H5Z_EDC_t check);
/**
* \ingroup DXPL
*
* \brief Sets user-defined filter callback function
*
* \dxpl_id{plist_id}
* \param[in] func User-defined filter callback function
* \param[in] op_data User-defined input data for the callback function
* \return \herr_t
*
* \details H5Pset_filter_callback() sets the user-defined filter callback
* function \p func in the dataset transfer property list \p plist_id.
*
* The parameter \p op_data is a pointer to user-defined input data for
* the callback function and will be passed through to the callback
* function.
*
* The callback function \p func defines the actions an application is
* to take when a filter fails. The function prototype is as follows:
* \snippet H5Zpublic.h H5Z_filter_func_t_snip
* where \c filter indicates which filter has failed, \c buf and \c buf_size
* are used to pass in the failed data, and op_data is the required
* input data for this callback function.
*
* Valid callback function return values are #H5Z_CB_FAIL and #H5Z_CB_CONT.
*
* \since 1.6.0
*
*/
H5_DLL herr_t H5Pset_filter_callback(hid_t plist_id, H5Z_filter_func_t func, void *op_data);
H5_DLL herr_t H5Pset_hyper_vector_size(hid_t fapl_id, size_t size);
/**
* \ingroup DXPL
*
* \brief Sets number of I/O vectors to be read/written in hyperslab I/O
*
* \dxpl_id{plist_id}
* \param[in] size Number of I/O vectors to accumulate in memory for I/O
* operations\n
* Must be greater than 1 (one)\n
* Default value: 1024
* \return \herr_t
*
* \details H5Pset_hyper_vector_size() sets the number of I/O vectors to be
* accumulated in memory before being issued to the lower levels of
* the HDF5 library for reading or writing the actual data.
*
* The I/O vectors are hyperslab offset and length pairs and are
* generated during hyperslab I/O.
*
* The number of I/O vectors is passed in \p size to be set in the
* dataset transfer property list \p plist_id. \p size must be
* greater than 1 (one).
*
* H5Pset_hyper_vector_size() is an I/O optimization function;
* increasing vector_size should provide better performance, but the
* library will use more memory during hyperslab I/O. The default value
* of \p size is 1024.
*
* \since 1.6.0
*
*/
H5_DLL herr_t H5Pset_hyper_vector_size(hid_t plist_id, size_t size);
/**
* \ingroup DXPL
*
* \brief Sets the dataset transfer property list \p status
*
* \dxpl_id{plist_id}
* \param[in] status Status toggle of the dataset transfer property list
* \return \herr_t
*
* \deprecated This function is deprecated as it no longer has any effect;
* compound datatype field preservation is now core functionality in
* the HDF5 library.
*
* \details H5Pset_preserve() sets the dataset transfer property list status to
* \c 1 or \c 0.
*
* When reading or writing compound datatypes and the destination is
* partially initialized and the read/write is intended to initialize
* the other members, one must set this property to \c 1. Otherwise the
* I/O pipeline treats the destination datapoints as completely
* uninitialized.
*
* \todo Add missing version information: introduction, deprecation, etc.
* Why is the declaration not in the deprecated section?
*
*/
H5_DLL herr_t H5Pset_preserve(hid_t plist_id, hbool_t status);
/**
* \ingroup DXPL
*
* \brief Sets user-defined datatype conversion callback function
*
* \dxpl_id
* \param[in] op User-defined type conversion callback function
* \param[in] operate_data User-defined input data for the callback function
* \return \herr_t
*
* \details H5Pset_type_conv_cb() sets the user-defined datatype conversion
* callback function \p op in the dataset transfer property list \p
* dxpl_id
*
* The parameter operate_data is a pointer to user-defined input data
* for the callback function and will be passed through to the callback
* function.
*
* The callback function \p op defines the actions an application is to
* take when there is an exception during datatype conversion. The
* function prototype is as follows:
* \snippet H5Tpublic.h H5T_conv_except_func_t_snip
*
* \todo Add version information.
*
*/
H5_DLL herr_t H5Pset_type_conv_cb(hid_t dxpl_id, H5T_conv_except_func_t op, void *operate_data);
/**
* \ingroup DXPL
*
* \brief Sets the memory manager for variable-length datatype allocation in
* H5Dread() and H5Dvlen_reclaim()
*
* \dxpl_id{plist_id}
* \param[in] alloc_func User's allocate routine, or \c NULL for system \c malloc
* \param[in] alloc_info Extra parameter for user's allocation routine.
* Contents are ignored if preceding parameter is \c NULL.
* \param[in] free_func User's free routine, or \c NULL for system \c free
* \param[in] free_info Extra parameter for user's free routine. Contents are
* ignored if preceding parameter is \c NULL
* \return \herr_t
*
* \details H5Pset_vlen_mem_manager() sets the memory manager for
* variable-length datatype allocation in H5Dread() and free in
* H5Dvlen_reclaim().
*
* The \p alloc_func and \p free_func parameters identify the memory
* management routines to be used. If the user has defined custom
* memory management routines, \p alloc_func and/or free_func should be
* set to make those routine calls (i.e., the name of the routine is
* used as the value of the parameter); if the user prefers to use the
* system's \c malloc and/or \c free, the \p alloc_func and \p
* free_func parameters, respectively, should be set to \c NULL
*
* The prototypes for these user-defined functions are as follows:
* \snippet H5MMpublic.h H5MM_allocate_t_snip
*
* \snippet H5MMpublic.h H5MM_free_t_snip
*
* The \p alloc_info and \p free_info parameters can be used to pass
* along any required information to the user's memory management
* routines.
*
* In summary, if the user has defined custom memory management
* routines, the name(s) of the routines are passed in the \p
* alloc_func and \p free_func parameters and the custom routines'
* parameters are passed in the \p alloc_info and \p free_info
* parameters. If the user wishes to use the system \c malloc and \c
* free functions, the \p alloc_func and/or \p free_func parameters are
* set to \c NULL and the \p alloc_info and \p free_info parameters are
* ignored.
*
* \todo Add version information.
*/
H5_DLL herr_t H5Pset_vlen_mem_manager(hid_t plist_id, H5MM_allocate_t alloc_func, void *alloc_info,
H5MM_free_t free_func, void *free_info);
#ifdef H5_HAVE_PARALLEL
/**
* \ingroup DXPL
*
* \brief Retrieves the type of chunk optimization that HDF5 actually performed
* on the last parallel I/O call (not necessarily the type requested)
*
* \dxpl_id{plist_id}
* \param[out] actual_chunk_opt_mode The type of chunk optimization performed by HDF5
* \return \herr_t
*
* \par Motivation:
* A user can request collective I/O via a data transfer property list
* (DXPL) that has been suitably modified with H5Pset_dxpl_mpio().
* However, HDF5 will sometimes ignore this request and perform independent
* I/O instead. This property allows the user to see what kind of I/O HDF5
* actually performed. Used in conjunction with H5Pget_mpio_actual_io_mode(),
* this property allows the user to determine exactly what HDF5 did when
* attempting collective I/O.
*
* \details H5Pget_mpio_actual_chunk_opt_mode() retrieves the type of chunk
* optimization performed when collective I/O was requested. This
* property is set before I/O takes place, and will be set even if I/O
* fails.
*
* Valid values returned in \p actual_chunk_opt_mode:
* \snippet this H5D_mpio_actual_chunk_opt_mode_t_snip
* \click4more
*
* \since 1.8.8
*
*/
H5_DLL herr_t H5Pget_mpio_actual_chunk_opt_mode(hid_t plist_id,
H5D_mpio_actual_chunk_opt_mode_t *actual_chunk_opt_mode);
/**
* \ingroup DXPL
*
* \brief Retrieves the type of I/O that HDF5 actually performed on the last
* parallel I/O call (not necessarily the type requested)
*
* \dxpl_id{plist_id}
* \param[out] actual_io_mode The type of I/O performed by this process
* \return \herr_t
*
* \par Motivation:
* A user can request collective I/O via a data transfer property list
* (DXPL) that has been suitably modified with H5Pset_dxpl_mpio().
* However, HDF5 will sometimes ignore this request and perform independent
* I/O instead. This property allows the user to see what kind of I/O HDF5
* actually performed. Used in conjunction with H5Pget_mpio_actual_chunk_opt_mode(),
* this property allows the user to determine exactly HDF5 did when
* attempting collective I/O.
*
* \details H5Pget_mpio_actual_io_mode() retrieves the type of I/O performed on
* the selection of the current process. This property is set after all
* I/O is completed; if I/O fails, it will not be set.
*
* Valid values returned in \p actual_io_mode:
* \snippet this H5D_mpio_actual_io_mode_t_snip
* \click4more
*
* \attention All processes do not need to have the same value. For example, if
* I/O is being performed using the multi chunk optimization scheme,
* one process's selection may include only chunks accessed
* collectively, while another may include chunks accessed
* independently. In this case, the first process will report
* #H5D_MPIO_CHUNK_COLLECTIVE while the second will report
* #H5D_MPIO_CHUNK_INDEPENDENT.
*
* \see H5Pget_mpio_no_collective_cause(), H5Pget_mpio_actual_chunk_opt_mode()
*
* \since 1.8.8
*
*/
H5_DLL herr_t H5Pget_mpio_actual_io_mode(hid_t plist_id, H5D_mpio_actual_io_mode_t *actual_io_mode);
/**
* \ingroup DXPL
*
* \brief Retrieves local and global causes that broke collective I/O on the last
* parallel I/O call
*
* \dxpl_id{plist_id}
* \param[out] local_no_collective_cause An enumerated set value indicating the
* causes that prevented collective I/O in the local process
* \param[out] global_no_collective_cause An enumerated set value indicating
* the causes across all processes that prevented collective I/O
* \return \herr_t
*
* \par Motivation:
* A user can request collective I/O via a data transfer property list (DXPL)
* that has been suitably modified with H5P_SET_DXPL_MPIO. However, there are
* conditions that can cause HDF5 to forgo collective I/O and perform
* independent I/O. Such causes can be different across the processes of a
* parallel application. This function allows the user to determine what
* caused the HDF5 library to skip collective I/O locally, that is in the
* local process, and globally, across all processes.
*
* \details H5Pget_mpio_no_collective_cause() serves two purposes. It can be
* used to determine whether collective I/O was used for the last
* preceding parallel I/O call. If collective I/O was not used, the
* function retrieves the local and global causes that broke collective
* I/O on that parallel I/O call. The properties retrieved by this
* function are set before I/O takes place and are retained even when
* I/O fails.
*
* Valid values returned in \p local_no_collective_cause and \p
* global_no_collective_cause are as follows or, if there are multiple
* causes, a bitwise OR of the relevant causes; the numbers in the
* center column are the bitmask values:
* \snippet this H5D_mpio_no_collective_cause_t_snip
* \click4more
*
* \attention Each process determines whether it can perform collective I/O and
* broadcasts the result. Those results are combined to make a
* collective decision; collective I/O will be performed only if all
* processes can perform collective I/O.\n
* If collective I/O was not used, the causes that prevented it are
* reported by individual process by means of an enumerated set. The
* causes may differ among processes, so H5Pget_mpio_no_collective_cause()
* returns two property values. The first value is the one produced
* by the local process to report local causes. This local information
* is encoded in an enumeration, the \ref H5D_mpio_no_collective_cause_t
* described above, with all individual causes combined into a single
* enumeration value by means of a bitwise OR operation. The second
* value reports global causes; this global value is the result of a
* bitwise-OR operation across the values returned by all the processes.
*
* \since 1.8.10
*
*/
H5_DLL herr_t H5Pget_mpio_no_collective_cause(hid_t plist_id, uint32_t *local_no_collective_cause,
uint32_t *global_no_collective_cause);
#endif /* H5_HAVE_PARALLEL */
@@ -5880,6 +8052,36 @@ H5_DLL herr_t H5Pget_create_intermediate_group(hid_t plist_id, unsigned *crt_int
H5_DLL herr_t H5Pset_create_intermediate_group(hid_t plist_id, unsigned crt_intmd);
/* Group creation property list (GCPL) routines */
/**
* \ingroup GCPL
*
* \brief Returns the estimated link count and average link name length in a group
*
* \gcpl_id{plist_id}
* \param[out] est_num_entries The estimated number of links in the group
* referenced by \p plist_id
* \param[out] est_name_len The estimated average length of line names in the group
* referenced by \p plist_id
* \return \herr_t
*
* \details H5Pget_est_link_info() retrieves two settings from the group creation
* property list \p plist_id: the estimated number of links that are
* expected to be inserted into a group created with the property list
* and the estimated average length of those link names.
*
* The estimated number of links is returned in \p est_num_entries. The
* limit for \p est_num_entries is 64 K.
*
* The estimated average length of the anticipated link names is returned
* in \p est_name_len. The limit for \p est_name_len is 64 K.
*
* See \ref_group_impls for a discussion of the available types of HDF5
* group structures.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pget_est_link_info(hid_t plist_id, unsigned *est_num_entries /* out */,
unsigned *est_name_len /* out */);
/**
@@ -5949,7 +8151,64 @@ H5_DLL herr_t H5Pget_link_creation_order(hid_t plist_id, unsigned *crt_order_fla
*/
H5_DLL herr_t H5Pget_link_phase_change(hid_t plist_id, unsigned *max_compact /*out*/,
unsigned *min_dense /*out*/);
/**
* \ingroup GCPL
*
* \brief Retrieves the anticipated size of the local heap for original-style
* groups
*
* \gcpl_id{plist_id}
* \param[out] size_hint Anticipated size of local heap
* \return \herr_t
*
* \details H5Pget_local_heap_size_hint() queries the group creation property
* list, \p plist_id, for the anticipated size of the local heap, \p
* size_hint, for original-style groups, i.e., for groups of the style
* used prior to HDF5 Release 1.8.0. See H5Pset_local_heap_size_hint()
* for further discussion.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pget_local_heap_size_hint(hid_t plist_id, size_t *size_hint /*out*/);
/**
* \ingroup GCPL
*
* \brief Sets estimated number of links and length of link names in a group
*
* \gcpl_id{plist_id}
* \param[in] est_num_entries Estimated number of links to be inserted into group
* \param[in] est_name_len Estimated average length of link names
* \return \herr_t
*
* \details H5Pset_est_link_info() inserts two settings into the group creation
* property list plist_id: the estimated number of links that are
* expected to be inserted into a group created with the property list
* and the estimated average length of those link names.
*
* The estimated number of links is passed in \p est_num_entries. The
* limit for \p est_num_entries is 64 K.
*
* The estimated average length of the anticipated link names is passed
* in \p est_name_len. The limit for \p est_name_len is 64 K.
*
* The values for these two settings are multiplied to compute the
* initial local heap size (for old-style groups, if the local heap
* size hint is not set) or the initial object header size for
* (new-style compact groups; see \ref_group_impls). Accurately setting
* these parameters will help reduce wasted file space.
*
* If a group is expected to have many links and to be stored in dense
* format, set \p est_num_entries to 0 (zero) for maximum
* efficiency. This will prevent the group from being created in the
* compact format.
*
* See \ref_group_impls for a discussion of the available types of HDF5
* group structures.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pset_est_link_info(hid_t plist_id, unsigned est_num_entries, unsigned est_name_len);
/**
* \ingroup GCPL
@@ -6043,11 +8302,115 @@ H5_DLL herr_t H5Pset_link_creation_order(hid_t plist_id, unsigned crt_order_flag
*
*/
H5_DLL herr_t H5Pset_link_phase_change(hid_t plist_id, unsigned max_compact, unsigned min_dense);
/**
* \ingroup GCPL
*
* \brief Specifies the anticipated maximum size of a local heap
*
* \gcpl_id{plist_id}
* \param[in] size_hint Anticipated maximum size in bytes of local heap
* \return \herr_t
*
* \details H5Pset_local_heap_size_hint() is used with original-style HDF5
* groups (see “Motivation” below) to specify the anticipated maximum
* local heap size, size_hint, for groups created with the group
* creation property list \p plist_id. The HDF5 library then uses \p
* size_hint to allocate contiguous local heap space in the file for
* each group created with \p plist_id.
*
* For groups with many members or very few members, an appropriate
* initial value of \p size_hint would be the anticipated number of
* group members times the average length of group member names, plus a
* small margin:
* \code
* size_hint = max_number_of_group_members *
* (average_length_of_group_member_link_names + 2)
* \endcode
* If it is known that there will be groups with zero members, the use
* of a group creation property list with \p size_hint set to to 1 (one)
* will guarantee the smallest possible local heap for each of those groups.
*
* Setting \p size_hint to zero (0) causes the library to make a
* reasonable estimate for the default local heap size.
*
* \par Motivation:
* In situations where backward-compatibility is required, specifically, when
* libraries prior to HDF5 Release 1.8.0 may be used to read the file, groups
* must be created and maintained in the original style. This is HDF5’s default
* behavior. If backward compatibility with pre-1.8.0 libraries is not a concern,
* greater efficiencies can be obtained with the new-format compact and indexed
* groups. See <a href="https://portal.hdfgroup.org/display/HDF5/Groups">Group
* implementations in HDF5</a> in the \ref H5G API introduction (at the bottom).\n
* H5Pset_local_heap_size_hint() is useful for tuning file size when files
* contain original-style groups with either zero members or very large
* numbers of members.\n
* The original style of HDF5 groups, the only style available prior to HDF5
* Release 1.8.0, was well-suited for moderate-sized groups but was not optimized
* for either very small or very large groups. This original style remains the
* default, but two new group implementations were introduced in HDF5 Release 1.8.0:
* compact groups to accommodate zero to small numbers of members and indexed groups
* for thousands or tens of thousands of members ... or millions, if that's what
* your application requires.\n
* The local heap size hint, \p size_hint, is a performance tuning parameter for
* original-style groups. As indicated above, an HDF5 group may have zero, a handful,
* or tens of thousands of members. Since the original style of HDF5 groups stores the
* metadata for all of these group members in a uniform format in a local heap, the size
* of that metadata (and hence, the size of the local heap) can vary wildly from group
* to group. To intelligently allocate space and to avoid unnecessary fragmentation of
* the local heap, it can be valuable to provide the library with a hint as to the local
* heap’s likely eventual size. This can be particularly valuable when it is known that
* a group will eventually have a great many members. It can also be useful in conserving
* space in a file when it is known that certain groups will never have any members.
*
* \since 1.8.0
*
*/
H5_DLL herr_t H5Pset_local_heap_size_hint(hid_t plist_id, size_t size_hint);
/* Map access property list (MAPL) routines */
#ifdef H5_HAVE_MAP_API
/**
* \ingroup MAPL
*
* \brief Set map iteration hints
*
* \mapl_id
* \param[in] key_prefetch_size Number of keys to prefetch at a time during
* iteration
* \param[in] key_alloc_size The initial size of the buffer allocated to hold
* prefetched keys
* \return \herr_t
*
* \details H5Pset_map_iterate_hints() adjusts the behavior of H5Miterate() when
* prefetching keys for iteration. The \p key_prefetch_size parameter
* specifies the number of keys to prefetch at a time during
* iteration. The \p key_alloc_size parameter specifies the initial
* size of the buffer allocated to hold these prefetched keys. If this
* buffer is too small it will be reallocated to a larger size, though
* this may result in an additional I/O.
*
* \since 1.12.?
*
*/
H5_DLL herr_t H5Pset_map_iterate_hints(hid_t mapl_id, size_t key_prefetch_size, size_t key_alloc_size);
/**
* \ingroup MAPL
*
* \brief Set map iteration hints
*
* \mapl_id
* \param[out] key_prefetch_size Pointer to number of keys to prefetch at a time
* during iteration
* \param[out] key_alloc_size Pointer to the initial size of the buffer allocated
* to hold prefetched keys
* \return \herr_t
*
* \details H5Pget_map_iterate() returns the map iterate hints, \p key_prefetch_size
* and \p key_alloc_size, as set by H5Pset_map_iterate_hints().
*
* \since 1.12.?
*
*/
H5_DLL herr_t H5Pget_map_iterate_hints(hid_t mapl_id, size_t *key_prefetch_size /*out*/,
size_t *key_alloc_size /*out*/);
#endif /* H5_HAVE_MAP_API */
@@ -6409,9 +8772,6 @@ H5_DLL herr_t H5Pset_elink_acc_flags(hid_t lapl_id, unsigned flags);
* </pre>
*
*
* \todo Add Programming Note for C++ Developers Using C Functions
*
*
* \since 1.8.3
*
*/
@@ -6594,10 +8954,7 @@ H5_DLL herr_t H5Pset_nlinks(hid_t plist_id, size_t nlinks);
* \li H5Pget_mcdt_search_cb()
* \li H5Pset_copy_object()
* \li H5Pset_mcdt_search_cb()
*
* \todo missing link to "Copying Committed Datatypes with H5Ocopy - A
* comprehensive discussion of copying committed datatypes (PDF)
* in Advanced Topics in HDF5
* \li \ref_h5ocopy
*
* \since 1.8.9
*
@@ -6711,8 +9068,7 @@ H5_DLL herr_t H5Pget_copy_object(hid_t plist_id, unsigned *copy_options /*out*/)
* \li H5Pget_mcdt_search_cb()
* \li H5Pset_copy_object()
* \li H5Pset_mcdt_search_cb()
*
* \todo Link to Copying Committed Datatypes with H5Ocopy was removed.
* \li \ref_h5ocopy
*
* \since 1.8.9
*
@@ -6803,8 +9159,8 @@ H5_DLL herr_t H5Pget_mcdt_search_cb(hid_t plist_id, H5O_mcdt_search_cb_t *func,
* \li H5Pget_mcdt_search_cb()
* \li H5Pset_copy_object()
* \li H5Pset_mcdt_search_cb()
* \li \ref_h5ocopy
*
* \todo Link to Copying Committed Datatypes with H5Ocopy was removed.
* \version 1.8.9 #H5O_COPY_MERGE_COMMITTED_DTYPE_FLAG added in this release.
*
* \since 1.8.0
@@ -6890,9 +9246,7 @@ H5_DLL herr_t H5Pset_copy_object(hid_t plist_id, unsigned copy_options);
* \li H5Pget_mcdt_search_cb()
* \li H5Pset_copy_object()
* \li H5Pset_mcdt_search_cb()
*
* \todo Link removed to "Copying Committed Datatypes with H5Ocopy" in Advanced Topics in HDF5
* \todo Programming Note for C++ Developers Using C Functions:
* \li \ref_h5ocopy
*
* \since 1.8.9
*
@@ -6911,38 +9265,316 @@ H5_DLL herr_t H5Pset_mcdt_search_cb(hid_t plist_id, H5O_mcdt_search_cb_t func, v
#define H5P_NO_CLASS H5P_ROOT
/* Typedefs */
/**
* \ingroup GPLOA
*
* \brief Registers a permanent property with a property list class
*
* \plistcls_id{cls_id}
* \param[in] name Name of property to register
* \param[in] size Size of property in bytes
* \param[in] def_value Default value for property in newly created
* property lists
* \param[in] prp_create Callback routine called when a property list is
* being created and the property value will be
* initialized
* \param[in] prp_set Callback routine called before a new value is
* copied into the property's value
* \param[in] prp_get Callback routine called when a property value is
* retrieved from the property
* \param[in] prp_del Callback routine called when a property is deleted
* from a property list
* \param[in] prp_copy Callback routine called when a property is copied
* from a property list
* \param[in] prp_close Callback routine called when a property list is
* being closed and the property value will be
* disposed of
*
* \return \herr_t
*
* \deprecated As of HDF5-1.8 this function was deprecated in favor of
* H5Pregister2() or the macro H5Pregister().
*
* \details H5Pregister1() registers a new property with a property list
* class. The property will exist in all property list objects
* of that class after this routine is finished. The name of
* the property must not already exist. The default property
* value must be provided and all new property lists created
* with this property will have the property value set to the
* default provided. Any of the callback routines may be set
* to NULL if they are not needed.
*
* Zero-sized properties are allowed and do not store any data in
* the property list. These may be used as flags to indicate the
* presence or absence of a particular piece of information. The
* default pointer for a zero-sized property may be set to NULL.
* The property \p prp_create and \p prp_close callbacks are called for
* zero-sized properties, but the \p prp_set and \p prp_get callbacks
* are never called.
*
* The \p prp_create routine is called when a new property list with
* this property is being created. The #H5P_prp_create_func_t
* callback function is defined as #H5P_prp_cb1_t.
*
* The \p prp_create routine may modify the value to be set and those
* changes will be stored as the initial value of the property.
* If the \p prp_create routine returns a negative value, the new
* property value is not copied into the property and the
* \p prp_create routine returns an error value.
*
* The \p prp_set routine is called before a new value is copied into
* the property. The #H5P_prp_set_func_t callback function is defined
* as #H5P_prp_cb2_t.
*
* The \p prp_set routine may modify the value pointer to be set and
* those changes will be used when setting the property's value.
* If the \p prp_set routine returns a negative value, the new property
* value is not copied into the property and the \p prp_set routine
* returns an error value. The \p prp_set routine will not be called
* for the initial value; only the \p prp_create routine will be
* called.
*
* \b Note: The \p prp_set callback function may be useful to range
* check the value being set for the property or may perform some
* transformation or translation of the value set. The \p prp_get
* callback would then reverse the transformation or translation.
* A single \p prp_get or \p prp_set callback could handle multiple
* properties by performing different actions based on the property
* name or other properties in the property list.
*
* The \p prp_get routine is called when a value is retrieved from a
* property value. The #H5P_prp_get_func_t callback function is
* defined as #H5P_prp_cb2_t.
*
* The \p prp_get routine may modify the value to be returned from the
* query and those changes will be returned to the calling routine.
* If the \p prp_set routine returns a negative value, the query
* routine returns an error value.
*
* The \p prp_del routine is called when a property is being
* deleted from a property list. The #H5P_prp_delete_func_t
* callback function is defined as #H5P_prp_cb2_t.
*
* The \p prp_del routine may modify the value passed in, but the
* value is not used by the library when the \p prp_del routine
* returns. If the \p prp_del routine returns a negative value,
* the property list deletion routine returns an error value but
* the property is still deleted.
*
* The \p prp_copy routine is called when a new property list with
* this property is being created through a \p prp_copy operation.
* The #H5P_prp_copy_func_t callback function is defined as
* #H5P_prp_cb1_t.
*
* The \p prp_copy routine may modify the value to be set and those
* changes will be stored as the new value of the property. If
* the \p prp_copy routine returns a negative value, the new
* property value is not copied into the property and the \p prp_copy
* routine returns an error value.
*
* The \p prp_close routine is called when a property list with this
* property is being closed. The #H5P_prp_close_func_t callback
* function is defined as #H5P_prp_cb1_t.
*
* The \p prp_close routine may modify the value passed in, but the
* value is not used by the library when the \p prp_close routine
* returns. If the \p prp_close routine returns a negative value, the
* property list close routine returns an error value but the property
* list is still closed.
*
* The #H5P_prp_cb1_t is as follows:
* \snippet this H5P_prp_cb1_t_snip
*
* The #H5P_prp_cb2_t is as follows:
* \snippet this H5P_prp_cb2_t_snip
*
*
* \cpp_c_api_note
*
*/
/* Function prototypes */
H5_DLL herr_t H5Pregister1(hid_t cls_id, const char *name, size_t size, void *def_value,
H5P_prp_create_func_t prp_create, H5P_prp_set_func_t prp_set,
H5P_prp_get_func_t prp_get, H5P_prp_delete_func_t prp_del,
H5P_prp_copy_func_t prp_copy, H5P_prp_close_func_t prp_close);
/**
* \ingroup GPLOA
*
* \brief Registers a temporary property with a property list
*
* \plist_id
* \param[in] name Name of property to create
* \param[in] size Size of property in bytes
* \param[in] value Initial value for the property
* \param[in] prp_set Callback routine called before a new value is copied
* into the property's value
* \param[in] prp_get Callback routine called when a property value is
* retrieved from the property
* \param[in] prp_delete Callback routine called when a property is deleted
* from a property list
* \param[in] prp_copy Callback routine called when a property is copied
* from an existing property list
* \param[in] prp_close Callback routine called when a property list is
* being closed and the property value will be disposed
* of
*
* \return \herr_t
*
* \deprecated As of HDF5-1.8 this function was deprecated in favor of
* H5Pinsert2() or the macro H5Pinsert().
*
* \details H5Pinsert1() creates a new property in a property
* list. The property will exist only in this property list and
* copies made from it.
*
* The initial property value must be provided in \p value and
* the property value will be set accordingly.
*
* The name of the property must not already exist in this list,
* or this routine will fail.
*
* The \p prp_set and \p prp_get callback routines may be set to NULL
* if they are not needed.
*
* Zero-sized properties are allowed and do not store any data
* in the property list. The default value of a zero-size
* property may be set to NULL. They may be used to indicate the
* presence or absence of a particular piece of information.
*
* The \p prp_set routine is called before a new value is copied
* into the property. The #H5P_prp_set_func_t callback function
* is defined as #H5P_prp_cb2_t.
* The \p prp_set routine may modify the value pointer to be set and
* those changes will be used when setting the property's value.
* If the \p prp_set routine returns a negative value, the new property
* value is not copied into the property and the \p set routine
* returns an error value. The \p prp_set routine will be called for
* the initial value.
*
* \b Note: The \p prp_set callback function may be useful to range
* check the value being set for the property or may perform some
* transformation or translation of the value set. The \p prp_get
* callback would then reverse the transformation or translation.
* A single \p prp_get or \p prp_set callback could handle multiple
* properties by performing different actions based on the
* property name or other properties in the property list.
*
* The \p prp_get routine is called when a value is retrieved from
* a property value. The #H5P_prp_get_func_t callback function
* is defined as #H5P_prp_cb2_t.
*
* The \p prp_get routine may modify the value to be returned from
* the query and those changes will be preserved. If the \p prp_get
* routine returns a negative value, the query routine returns
* an error value.
*
* The \p prp_delete routine is called when a property is being
* deleted from a property list. The #H5P_prp_delete_func_t
* callback function is defined as #H5P_prp_cb2_t.
*
* The \p prp_copy routine is called when a new property list with
* this property is being created through a \p prp_copy operation.
* The #H5P_prp_copy_func_t callback function is defined as
* #H5P_prp_cb1_t.
*
* The \p prp_copy routine may modify the value to be set and those
* changes will be stored as the new value of the property. If the
* \p prp_copy routine returns a negative value, the new property value
* is not copied into the property and the prp_copy routine returns an
* error value.
*
* The \p prp_close routine is called when a property list with this
* property is being closed.
* The #H5P_prp_close_func_t callback function is defined as
* #H5P_prp_cb1_t.
*
* The \p prp_close routine may modify the value passed in, the
* value is not used by the library when the close routine
* returns. If the \p prp_close routine returns a negative value,
* the property list \p prp_close routine returns an error value
* but the property list is still closed.
*
* \b Note: There is no \p prp_create callback routine for temporary
* property list objects; the initial value is assumed to
* have any necessary setup already performed on it.
*
* The #H5P_prp_cb1_t is as follows:
* \snippet this H5P_prp_cb1_t_snip
*
* The #H5P_prp_cb2_t is as follows:
* \snippet this H5P_prp_cb2_t_snip
* \cpp_c_api_note
*/
H5_DLL herr_t H5Pinsert1(hid_t plist_id, const char *name, size_t size, void *value,
H5P_prp_set_func_t prp_set, H5P_prp_get_func_t prp_get,
H5P_prp_delete_func_t prp_delete, H5P_prp_copy_func_t prp_copy,
H5P_prp_close_func_t prp_close);
/**
* \ingroup GPLO
*
* \brief Encodes the property values in a property list into a binary
* buffer
*
* \plist_id
* \param[out] buf Buffer into which the property list will be encoded.
* If the provided buffer is NULL, the size of the
* buffer required is returned through \p nalloc; the
* function does nothing more.
* \param[out] nalloc The size of the required buffer
*
* \return \herr_t
*
* \deprecated As of HDF5-1.12 this function has been deprecated in favor of
* H5Pencode2() or the macro H5Pencode().
*
* \details H5Pencode1() encodes the property list \p plist_id into the
* binary buffer \p buf.
*
* If the required buffer size is unknown, \p buf can be passed
* in as NULL and the function will set the required buffer size
* in \p nalloc. The buffer can then be created and the property
* list encoded with a subsequent H5Pencode1() call.
*
* If the buffer passed in is not big enough to hold the encoded
* properties, the H5Pencode1() call can be expected to fail with
* a segmentation fault.
*
* Properties that do not have encode callbacks will be skipped.
* There is currently no mechanism to register an encode callback for
* a user-defined property, so user-defined properties cannot currently
* be encoded.
*
* Some properties cannot be encoded, particularly properties that are
* reliant on local context.
*
* \since 1.10.0
*
*/
H5_DLL herr_t H5Pencode1(hid_t plist_id, void *buf, size_t *nalloc);
/**
* \ingroup OCPL
* \ingroup DCPL
*
* \brief Returns information about a filter in a pipeline (DEPRECATED)
*
* \todo H5Pget_filter1() prototype does not match source in H5Pocpl.c.
* Also, it is not in a deprecated file. Is that okay?
*
*
* \plist_id{plist_id}
* \param[in] filter Sequence number within the filter pipeline of the filter
* for which information is sought
* \param[out] flags Bit vector specifying certain general properties of
* the filter
* \param[in] filter Sequence number within the filter pipeline of
* the filter for which information is sought
* \param[out] flags Bit vector specifying certain general properties
* of the filter
* \param[in,out] cd_nelmts Number of elements in \p cd_values
* \param[out] cd_values Auxiliary data for the filter
* \param[in] namelen Anticipated number of characters in \p name
* \param[out] name Name of the filter
* \param[out] cd_values Auxiliary data for the filter
* \param[in] namelen Anticipated number of characters in \p name
* \param[out] name Name of the filter
*
* \return Returns the filter identifier if successful; Otherwise returns
* a negative value. See: #H5Z_filter_t
*
* \deprecated When was this function deprecated?
*
* \details H5Pget_filter1() returns information about a filter, specified
* by its filter number, in a filter pipeline, specified by the
* property list with which it is associated.
@@ -6977,13 +9609,127 @@ H5_DLL herr_t H5Pencode1(hid_t plist_id, void *buf, size_t *nalloc);
H5_DLL H5Z_filter_t H5Pget_filter1(hid_t plist_id, unsigned filter, unsigned int *flags /*out*/,
size_t *cd_nelmts /*out*/, unsigned cd_values[] /*out*/, size_t namelen,
char name[]);
H5_DLL herr_t H5Pget_filter_by_id1(hid_t plist_id, H5Z_filter_t id, unsigned int *flags /*out*/,
size_t *cd_nelmts /*out*/, unsigned cd_values[] /*out*/, size_t namelen,
char name[] /*out*/);
H5_DLL herr_t H5Pget_version(hid_t plist_id, unsigned *boot /*out*/, unsigned *freelist /*out*/,
unsigned *stab /*out*/, unsigned *shhdr /*out*/);
H5_DLL herr_t H5Pset_file_space(hid_t plist_id, H5F_file_space_type_t strategy, hsize_t threshold);
H5_DLL herr_t H5Pget_file_space(hid_t plist_id, H5F_file_space_type_t *strategy, hsize_t *threshold);
/**
* \ingroup DCPL
*
* \brief Returns information about the specified filter
*
* \plist_id{plist_id}
* \param[in] id Filter identifier
* \param[out] flags Bit vector specifying certain general properties
* of the filter
* \param[in,out] cd_nelmts Number of elements in \p cd_values
* \param[out] cd_values Auxiliary data for the filter
* \param[in] namelen Anticipated number of characters in \p name
* \param[out] name Name of the filter
*
*
* \return Returns a non-negative value if successful; Otherwise returns
* a negative value.
*
* \deprecated As of HDF5-1.8 this function was deprecated in favor of
* H5Pget_filter_by_id2() or the macro H5Pget_filter_by_id().
*
* \details H5Pget_filter_by_id1() returns information about a filter, specified
* in \p id, a filter identifier.
*
* \p plist_id must be a dataset or group creation property list and
* \p id must be in the associated filter pipeline.
*
* The \p id and \p flags parameters are used in the same
* manner as described in the discussion of H5Pset_filter().
*
* Aside from the fact that they are used for output, the parameters
* \p cd_nelmts and \p cd_values[] are used in the same manner as
* described in the discussion of H5Pset_filter().
* On input, the \p cd_nelmts parameter indicates the number of entries
* in the \p cd_values[] array allocated by the calling program;
* on exit it contains the number of values defined by the filter.
*
* On input, the \p namelen parameter indicates the number of
* characters allocated for the filter name by the calling program
* in the array \p name[]. On exit \p name[] contains the name of the
* filter with one character of the name in each element of the array.
*
* If the filter specified in \p id is not set for the property
* list, an error will be returned and this function will fail.
*
*
* \version 1.8.5 Function extended to work with group creation property
* lists.
* \version 1.8.0 Function H5Pget_filter_by_id() renamed to
* H5Pget_filter_by_id1() and deprecated in this release.
* \version 1.6.0 Function introduced in this release.
*/
H5_DLL herr_t H5Pget_filter_by_id1(hid_t plist_id, H5Z_filter_t id, unsigned int *flags /*out*/,
size_t *cd_nelmts /*out*/, unsigned cd_values[] /*out*/, size_t namelen,
char name[] /*out*/);
/**
* \ingroup FCPL
*
* \brief Retrieves the version information of various objects
* for a file creation property list(deprecated)
*
* \plist_id
* \param[out] boot Pointer to location to return super block version number
* \param[out] freelist Pointer to location to return global freelist version number
* \param[out] stab Pointer to location to return symbol table version number
* \param[out] shhdr Pointer to location to return shared object header version
* number
*
* \return \herr_t
*
* \deprecated Deprecated in favor of the function H5Fget_info()
*
* \details H5Pget_version() retrieves the version information of various objects
* for a file creation property list. Any pointer parameters which are
* passed as NULL are not queried.
*
* \version 1.6.4 \p boot, \p freelist, \p stab, \p shhdr parameter types
* changed to unsigned.
*
*/
H5_DLL herr_t H5Pget_version(hid_t plist_id, unsigned *boot /*out*/, unsigned *freelist /*out*/,
unsigned *stab /*out*/, unsigned *shhdr /*out*/);
/**
* \ingroup FCPL
*
* \brief Sets the file space handling strategy and the free-space section
* size threshold.
*
* \fcpl_id{plist_id}
* \param[in] strategy The file space handling strategy to be used. See:
* #H5F_fspace_strategy_t
* \param[in] threshold The smallest free-space section size that the free
* space manager will track
*
* \return \herr_t
*
* \deprecated When was this function deprecated?
*
* \details Maps to the function H5Pset_file_space_strategy().
*
*/
H5_DLL herr_t H5Pset_file_space(hid_t plist_id, H5F_file_space_type_t strategy, hsize_t threshold);
/**
* \ingroup FCPL
*
* \brief Retrieves the file space handling strategy, and threshold value for
* a file creation property list
*
* \fcpl_id{plist_id}
* \param[out] strategy Pointer to the file space handling strategy
* \param[out] threshold Pointer to the free-space section size threshold value
*
* \return \herr_t
*
* \deprecated When was this function deprecated?
*
* \details Maps to the function H5Pget_file_space_strategy()
*
*
*/
H5_DLL herr_t H5Pget_file_space(hid_t plist_id, H5F_file_space_type_t *strategy, hsize_t *threshold);
#endif /* H5_NO_DEPRECATED_SYMBOLS */
#ifdef __cplusplus

Some files were not shown because too many files have changed in this diff Show More