mirror of
https://github.com/HDFGroup/hdf5.git
synced 2026-09-25 04:09:44 +03:00
vchoi-hdfgroup
3e023fa22d
Issue #6616: Group Info and Link Info are labeled Optional, but new-style group creation always writes both, and the C library errors reading a new-style group missing either one. - Group Info / Link Info Status: Optional -> Required for "new style" groups; may not be repeated (matches existing Symbol Table wording for old-style groups). - Documents the decoder's fallback values (8, 6) for Link Phase Change: Maximum Compact/Minimum Dense Value when absent.
13383 lines
462 KiB
Plaintext
13383 lines
462 KiB
Plaintext
|
|
/** \page FMT4 HDF5 File Format Specification Version 4.0
|
|
|
|
Navigate back: \ref index "Main" / \ref SPEC
|
|
<hr>
|
|
|
|
<table align="right">
|
|
<tr><td>
|
|
<ol type="I">
|
|
<li> @ref sec_fmt4_intro
|
|
<ol type="A">
|
|
<li> @ref subsec_fmt4_intro_doc</li>
|
|
<li> @ref subsec_fmt4_intro_20</li>
|
|
<li> @ref subsec_fmt4_intro_112</li>
|
|
<li> @ref subsec_fmt4_intro_110</li>
|
|
</ol></li>
|
|
<li> @ref sec_fmt4_meta
|
|
<ol type="A">
|
|
<li> @ref subsec_fmt4_boot_super</li>
|
|
<li> @ref subsec_fmt4_boot_driver</li>
|
|
<li> @ref subsec_fmt4_boot_supext</li>
|
|
</ol></li>
|
|
<li> @ref sec_fmt4_infra
|
|
<ol type="A">
|
|
<li> @ref subsec_fmt4_infra_btrees
|
|
<ol type="1">
|
|
<li> @ref subsubsec_fmt4_infra_btrees_v1</li>
|
|
<li> @ref subsubsec_fmt4_infra_btrees_v2</li>
|
|
</ol></li>
|
|
<li> @ref subsec_fmt4_infra_symboltable</li>
|
|
<li> @ref subsec_fmt4_infra_symboltableentry</li>
|
|
<li> @ref subsec_fmt4_infra_localheap</li>
|
|
<li> @ref subsec_fmt4_infra_globalheap</li>
|
|
<li> @ref subsec_fmt4_infra_globalheapvds</li>
|
|
<li> @ref subsec_fmt4_infra_fractalheap</li>
|
|
<li> @ref subsec_fmt4_infra_freespaceindex</li>
|
|
<li> @ref subsec_fmt4_infra_sohm</li>
|
|
<li> @ref subsec_fmt4_infra_mdci</li>
|
|
</ol></li>
|
|
<li> @ref sec_fmt4_dataobject
|
|
<ol type="A">
|
|
<li> @ref subsec_fmt4_dataobject_hdr
|
|
<ol type="1">
|
|
<li> @ref subsec_fmt4_dataobject_hdr_prefix</li>
|
|
<ol type="a">
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_prefix_one</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_prefix_two</li>
|
|
</ol></li>
|
|
<li> @ref subsec_fmt4_dataobject_hdr_msg</li>
|
|
<li> @ref subsec_fmt4_dataobject_hdr_msg_catalog</li>
|
|
<ol type="a">
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_nil</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_simple</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_linkinfo</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_dtmessage</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_ofvmessage</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_fvmessage</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_link</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_external</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_layout</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_bogus</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_groupinfo</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_filter</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_attribute</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_comment</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_omodified</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_shared</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_continuation</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_stmgroup</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_mod</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_btreek</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_drvinfo</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_attrinfo</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_refcount</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_fsinfo</li>
|
|
<li> @ref subsubsec_fmt4_dataobject_hdr_msg_mdci</li>
|
|
</ol></li>
|
|
</ol></li>
|
|
<li> @ref subsec_fmt4_dataobject_storage</li>
|
|
</ol>
|
|
</li>
|
|
<li> @ref sec_fmt4_appendixa
|
|
<li> @ref sec_fmt4_appendixb
|
|
<li> @ref sec_fmt4_appendixc
|
|
<ol type="A">
|
|
<li> @ref subsec_fmt4_appendixc_chunk
|
|
<li> @ref subsec_fmt4_appendixc_implicit
|
|
<li> @ref subsec_fmt4_appendixc_fixedarr
|
|
<li> @ref subsec_fmt4_appendixc_extarr
|
|
<li> @ref subsec_fmt4_appendixc_appv2btree
|
|
</ol>
|
|
</li>
|
|
<li> @ref sec_fmt4_appendixd
|
|
<ol type="A">
|
|
<li> @ref subsec_fmt4_appendixd_encode
|
|
<li> @ref subsec_fmt4_appendixd_encodet
|
|
<li> @ref subsec_fmt4_appendixd_encoderv
|
|
<li> @ref subsec_fmt4_appendixd_encodedp
|
|
</ol>
|
|
</li>
|
|
</ol>
|
|
</td></tr>
|
|
</table>
|
|
|
|
\section sec_fmt4_intro I. Introduction
|
|
|
|
<table align="right">
|
|
<tr>
|
|
<td><strong>Figure 1:</strong> Relationships among the HDF5 root group, other groups, and objects</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\image html FF-IH_FileGroup.gif</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Figure 2:</strong> HDF5 objects -- datasets, datatypes, or dataspaces</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\image html FF-IH_FileObject.gif</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The format of an HDF5 file on disk encompasses several key ideas of the HDF4 and AIO file formats as well
|
|
as addressing some shortcomings therein. The new format is more self-describing than the HDF4 format and
|
|
is more uniformly applied to data objects in the file.
|
|
|
|
An HDF5 file appears to the user as a directed graph. The nodes of this graph are the higher-level HDF5
|
|
objects that are exposed by the HDF5 APIs:
|
|
\li Groups
|
|
\li Datasets
|
|
\li Committed (formerly Named) datatypes
|
|
|
|
At the lowest level, as information is actually written to the disk, an HDF5 file is made up of the
|
|
following objects:
|
|
\li A superblock
|
|
\li B-tree nodes
|
|
\li Heap blocks
|
|
\li Object headers
|
|
\li Object data
|
|
\li Free space
|
|
|
|
The HDF5 library uses these lower-level objects to represent the higher-level objects that are then
|
|
presented to the user or to applications through the APIs. For instance, a group is an object header that
|
|
contains a message that points to a local heap (for storing the links to objects in the group) and to a
|
|
B-tree (which indexes the links). A dataset is an object header that contains messages that describe
|
|
datatype, dataspace, layout, filters, external files, fill value, and other elements with the layout message
|
|
pointing to either a raw data chunk or to a B-tree that points to raw data chunks.
|
|
|
|
\subsection subsec_fmt4_intro_doc I.A. This Document
|
|
This document describes the lower-level data objects; the higher-level objects and their properties are
|
|
described in the \ref UG.
|
|
|
|
Three levels of information comprise the file format. Level 0 contains basic information for identifying
|
|
and defining information about the file. Level 1 information contains the information about the pieces of a
|
|
file shared by many objects in the file (such as a B-trees and heaps). Level 2 is the rest of the file and
|
|
contains all of the data objects with each object partitioned into header information, also known as
|
|
<em>metadata</em>, and data.
|
|
|
|
The various components of the lower-level data objects are described in pairs of tables. The first table
|
|
shows the format layout, and the second table describes the fields. The titles of format layout tables
|
|
begin with “Layout”. The titles of the tables where the fields are described begin with
|
|
“Fields”. For example, the table that describes the format of the
|
|
@ref subsubsec_fmt4_infra_btrees_v2 has a title of “Layout: Version 2 B-tree Header”, and the
|
|
fields in the version 2 B-tree header are described in the table titled “Fields: Version 2 B-tree Header”.
|
|
|
|
The sizes of various fields in the following layout tables are determined by looking at the number of
|
|
columns the field spans in the table. There are exceptions:
|
|
\li The size may be overridden by specifying a size in parentheses
|
|
\li The size of addresses is determined by the <em> @ref FMT4SizeOfOffsetsV0 "Size of Offsets"</em> field
|
|
in the superblock and is indicated in this document with a superscripted ‘O’
|
|
\li The size of length fields is determined by the <em> @ref FMT4SizeOfLengthsV0 "Size of Lengths"</em> field
|
|
in the superblock and is indicated in this document with a superscripted ‘L’.
|
|
|
|
Values for all fields in this document should be treated as unsigned integers, unless otherwise noted in
|
|
the description of a field. Additionally, all metadata fields are stored in little-endian byte order.
|
|
|
|
All checksums used in the format are computed with the
|
|
<a href="https://www.burtleburtle.net/bob/hash/doobs.html">Jenkins’ lookup3</a> algorithm.
|
|
|
|
Whenever a bit flag or field is mentioned for an entry, bits are numbered from the lowest bit position
|
|
in the entry.
|
|
|
|
Various format tables in this document have cells with “This space inserted only to align table nicely”.
|
|
These entries in the table are just to make the table presentation nicer and do not represent any values
|
|
or padding in the file.
|
|
|
|
\subsection subsec_fmt4_intro_20 I.B. Changes for HDF5 2.0
|
|
The following sections have been changed or added for the 2.0 release:
|
|
\li Under @ref subsubsec_fmt4_dataobject_hdr_msg_dtmessage, in the Description for
|
|
“Fields:Datatype Message”, version 5 was added, as well as the new Complex class (11).
|
|
|
|
\subsection subsec_fmt4_intro_112 I.C. Changes for HDF5 1.12
|
|
The following sections have been changed or added for the 1.12 release:
|
|
\li Under @ref subsubsec_fmt4_dataobject_hdr_msg_dtmessage, in the Description for
|
|
“Fields:Datatype Message”, version 4 was added and Reference class (7) of the
|
|
datatype was updated to describe version 4.
|
|
\li @ref sec_fmt4_appendixd was added.
|
|
|
|
\subsection subsec_fmt4_intro_110 I.D. Changes for HDF5 1.10
|
|
The following sections have been changed or added for the 1.10 release:
|
|
\li In the @ref subsec_fmt4_boot_super section, version 3 of the superblock was added.
|
|
\li In the @ref subsec_fmt4_boot_supext section, a link to the Data Storage message was added.
|
|
\li In the @ref subsubsec_fmt4_infra_btrees_v2 section, additional B-tree types were added.
|
|
Tables that describe the @ref FMT4V2BtType10"type 10" and @ref FMT4V2BtType11"11" record
|
|
layouts were added at the end of the section.
|
|
\li The @ref subsec_fmt4_infra_globalheapvds was added.
|
|
\li @ref subsubsec_fmt4_dataobject_hdr_msg_layout section was changed. The name was changed,
|
|
and @ref FMT4DataLayoutV4"version 4" of the data layout message was added for the virtual type.
|
|
\li The @ref subsubsec_fmt4_dataobject_hdr_msg_fsinfo header message type was added.
|
|
\li The @ref subsubsec_fmt4_dataobject_hdr_msg_mdci header message type was added.
|
|
\li @ref sec_fmt4_appendixc was added. Five indexing types were added.
|
|
|
|
\section sec_fmt4_meta II. Disk Format: Level 0 - File Metadata
|
|
|
|
\subsection subsec_fmt4_boot_super II.A. Disk Format: Level 0A - Format Signature and Superblock
|
|
The superblock may begin at certain predefined offsets within the HDF5 file, allowing a block of
|
|
unspecified content for users to place additional information at the beginning (and end) of the HDF5 file
|
|
without limiting the HDF5 library’s ability to manage the objects within the file itself. This feature
|
|
was designed to accommodate wrapping an HDF5 file in another file format or adding descriptive information
|
|
to an HDF5 file without requiring the modification of the actual file’s information. The superblock
|
|
is located by searching for the HDF5 file signature at byte offset 0, byte offset 512 and at successive
|
|
locations in the file, each a multiple of two of the previous location, in other words, at these byte
|
|
offsets: 0, 512, 1024, 2048, and so on.
|
|
|
|
The superblock is composed of the format signature, followed by a superblock version number and information
|
|
that is specific to each version of the superblock.
|
|
|
|
Currently, there are four versions of the superblock format:
|
|
\li Version 0 is the default format.
|
|
\li Version 1 is the same as version 0 but with the “<em>Indexed Storage Internal Node K</em>”
|
|
field for storing non-default B-tree ‘K’ value.
|
|
\li Version 2 has some fields eliminated and compressed from superblock format versions 0 and 1. It has
|
|
added checksum support and superblock extension to store additional superblock metadata.
|
|
\li Version 3 is the same as version 2 except that the field “<em>File Consistency Flags</em>”
|
|
is used for file locking. This format version will enable support for the latest version.
|
|
|
|
Version 0 and 1 of the superblock are described below:
|
|
<table>
|
|
<caption align="top"><strong>Layout: Superblock (Versions 0 and 1)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Format Signature <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version \# of Superblock</td>
|
|
<td>Version \# of File’s Free Space Storage</td>
|
|
<td>Version \# of Root Group Symbol Table Entry</td>
|
|
<td>Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version \# of Shared Header Message Format</td>
|
|
<td>Size of Offsets</td>
|
|
<td>Size of Lengths</td>
|
|
<td>Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Group Leaf Node K</td>
|
|
<td colspan="2">Group Internal Node K</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">File Consistency Flags</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Indexed Storage Internal Node K<sup>1</sup></td>
|
|
<td colspan="2">Reserved <em>(zero)</em><sup>1</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Base Address<sup>O</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Address of File Free Space Info<sup>O</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">End of File Address<sup>O</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Driver Information Block Address<sup>O</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Root Group Symbol Table Entry</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘1’ in the above table are new in version 1 of the superblock.
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Superblock (Versions 0 and 1)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Format Signature</td>
|
|
<td>This field contains a constant value and can be used to quickly identify a file as being an HDF5
|
|
file. The constant value is designed to allow easy identification of an HDF5 file and to allow
|
|
certain types of data corruption to be detected. The file signature of an HDF5 file always
|
|
contains the following values:
|
|
<br /><br /><center>
|
|
<table align="center" cellpadding="4" width="100%">
|
|
<tr align="center">
|
|
<td align="right">Decimal:</td>
|
|
<td width="8%">137</td>
|
|
<td width="8%">72</td>
|
|
<td width="8%">68</td>
|
|
<td width="8%">70</td>
|
|
<td width="8%">13</td>
|
|
<td width="8%">10</td>
|
|
<td width="8%">26</td>
|
|
<td width="8%">10</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td align="right">Hexadecimal:</td>
|
|
<td width="8%">89</td>
|
|
<td width="8%">48</td>
|
|
<td width="8%">44</td>
|
|
<td width="8%">46</td>
|
|
<td width="8%">0d</td>
|
|
<td width="8%">0a</td>
|
|
<td width="8%">1a</td>
|
|
<td width="8%">0a</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td align="right">ASCII C Notation:</td>
|
|
<td width="8%">\211</td>
|
|
<td width="8%">H</td>
|
|
<td width="8%">D</td>
|
|
<td width="8%">F</td>
|
|
<td width="8%">\\r</td>
|
|
<td width="8%">\\n</td>
|
|
<td width="8%">\032</td>
|
|
<td width="8%">\\n</td>
|
|
</tr>
|
|
</table></center>
|
|
<br />
|
|
This signature both identifies the file as an HDF5 file and provides for immediate detection of common
|
|
file-transfer problems. The first two bytes distinguish HDF5 files on systems that expect the first two
|
|
bytes to identify the file type uniquely. The first byte is chosen as a non-ASCII value to reduce the
|
|
probability that a text file may be misrecognized as an HDF5 file; also, it catches bad file transfers
|
|
that clear bit 7. Bytes two through four name the format. The CR-LF sequence catches bad file transfers
|
|
that alter newline sequences. The control-Z character stops file display under MS-DOS. The final line
|
|
feed checks for the inverse of the CR-LF translation problem. (This is a direct descendent of the
|
|
<a href="http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.PNG-file-signature">PNG</a>
|
|
file signature.)<br />
|
|
<em>This field is present in version 0+ of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version Number of the Superblock</td>
|
|
<td>This value is used to determine the format of the information in the superblock. When the format of
|
|
the information in the superblock is changed, the version number is incremented to the next integer
|
|
and can be used to determine how the information in the superblock is formatted.<br />
|
|
Values of 0, 1 and 2 are defined for this field (the format of version 2 is described below, not
|
|
here).<br />
|
|
<em>This field is present in version 0+ of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version Number of the File’s Free Space Information</td>
|
|
<td>This value is used to determine the format of the file’s free space information.<br />
|
|
The only value currently valid in this field is ‘0’, which indicates that the file’s
|
|
free space index is as described in @ref subsec_fmt4_infra_freespaceindex below.<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version Number of the Root Group Symbol Table Entry</td>
|
|
<td>This value is used to determine the format of the information in the Root Group Symbol Table Entry.
|
|
When the format of the information in that field is changed, the version number is incremented to the
|
|
next integer and can be used to determine how the information in the field is formatted.<br />
|
|
The only value currently valid in this field is ‘0’, which indicates that the root group
|
|
symbol table entry is formatted as described in @ref subsec_fmt4_infra_symboltableentry below.<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version Number of the Shared Header Message Format</td>
|
|
<td>This value is used to determine the format of the information in a shared object header message.
|
|
Since the format of the shared header messages differs from the other private header messages, a
|
|
version number is used to identify changes in the format.<br />
|
|
The only value currently valid in this field is ‘0’, which indicates that shared
|
|
header messages are formatted as described in @ref subsubsec_fmt4_dataobject_hdr_msg_shared below.<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>\anchor FMT4SizeOfOffsetsV0 Size of Offsets</td>
|
|
<td>This value contains the number of bytes used to store addresses in the file. The values for the
|
|
addresses of objects in the file are offsets relative to a base address, usually the address of the
|
|
superblock signature. This allows a wrapper to be added after the file is created without invalidating
|
|
the internal offset locations.<br />
|
|
<em>This field is present in version 0+ of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>\anchor FMT4SizeOfLengthsV0 Size of Lengths</td>
|
|
<td>This value contains the number of bytes used to store the size of an object.<br />
|
|
<em>This field is present in version 0+ of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Group Leaf Node K</td>
|
|
<td>Each leaf node of a group B-tree will have at least this many entries but not more than twice this
|
|
many. If a group has a single leaf node then it may have fewer entries.<br />
|
|
This value must be greater than zero.<br />
|
|
See the @ref subsec_fmt4_infra_btrees below.<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Group Internal Node K</td>
|
|
<td>Each internal node of a group B-tree will have at least this many entries but not more than twice this
|
|
many. If the group has only one internal node then it might have fewer entries.<br />
|
|
This value must be greater than zero.<br />
|
|
See the @ref subsec_fmt4_infra_btrees below.<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>File Consistency Flags</td>
|
|
<td>This field is unused and should be ignored.<br />
|
|
<em>This field is present in version 0+ of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Indexed Storage Internal Node K</td>
|
|
<td>Each internal node of a indexed storage B-tree will have at least this many entries but not more than
|
|
twice this many. If the ndex storage B-tree has only one internal node then it might have fewer
|
|
entries.<br />
|
|
This value must be greater than zero.<br />
|
|
See the @ref subsec_fmt4_infra_btrees below.<br />
|
|
<em>This field is present in version 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Base Address</td>
|
|
<td>This is the absolute file address of the first byte of the HDF5 data within the file. The library
|
|
currently constrains this value to be the absolute file address of the superblock itself when creating
|
|
new files; future versions of the library may provide greater flexibility. When opening an existing
|
|
file and this address does not match the offset of the superblock, the library assumes that the entire
|
|
contents of the HDF5 file have been adjusted in the file and adjusts the base address and end of file
|
|
address to reflect their new positions in the file. Unless otherwise noted, all other file addresses
|
|
are relative to this base address.<br />
|
|
<em>This field is present in version 0+ of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Address of Global Free Space Index</td>
|
|
<td>The file’s free space management is not persistent for version 0 and 1 of the superblock.
|
|
Currently this field always contains the @ref FMT4UndefinedAddress "undefined address".<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>End of File Address</td>
|
|
<td>This is the absolute file address of the first byte past the end of all HDF5 data. It is used to
|
|
determine whether a file has been accidentally truncated and as an address where file data allocation
|
|
can occur if space from the free list is not used.<br />
|
|
<em>This field is present in version 0+ of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Driver Information Block Address</td>
|
|
<td>This is the relative file address of the file driver information block which contains driver-specific
|
|
information needed to reopen the file. If there is no driver information block then this entry should
|
|
be the @ref FMT4UndefinedAddress "undefined address".<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Root Group Symbol Table Entry</td>
|
|
<td>This is the @ref subsec_fmt4_infra_symboltableentry of the root group, which serves as the entry-point
|
|
into the group graph for the file.<br />
|
|
<em>This field is present in version 0 and 1 of the superblock.</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
Versions 2 and 3 of the superblock is described below:
|
|
<table>
|
|
<caption align="top"><strong>Layout: Superblock (Versions 2 and 3)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Format Signature <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version \# of Superblock</td>
|
|
<td>Size of Offsets</td>
|
|
<td>Size of Lengths</td>
|
|
<td>File Consistency Flags</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Base Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Superblock Extension Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />End of File Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Root Group Object Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Superblock Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Superblock (Versions 2 and 3)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Format Signature</td>
|
|
<td>This field is the same as described for versions 0 and 1 of the superblock.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version Number of the Superblock</td>
|
|
<td>This field has a value of 2 and has the same meaning as for versions 0 and 1.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Offsets</td>
|
|
<td>This field is the same as described for @ref FMT4SizeOfOffsetsV0 "versions 0 and 1" of the superblock.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Lengths</td>
|
|
<td>This field is the same as described for @ref FMT4SizeOfLengthsV0 "versions 0 and 1" of the superblock.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>File Consistency Flags</td>
|
|
<td>For superblock version 2: This field is unused and should be ignored.<br />
|
|
For superblock version 3: This value contains flags to ensure file consistency for file locking.
|
|
Currently, the following bit flags are defined:
|
|
<ul>
|
|
<li>Bit 0 if set indicates that the file has been opened for write access.</li>
|
|
<li>Bit 1 is reserved for future use.</li>
|
|
<li>Bit 2 if set indicates that the file has been opened for single-writer/multiple-reader
|
|
(SWMR) write access.</li>
|
|
<li>Bits 3-7 are reserved for future use.</li>
|
|
</ul><br />
|
|
Bit 0 should be set as the first action when a file has been opened for write access. Bit 2 should
|
|
be set when a file has been opened for SWMR write access. These two bits should be cleared only as
|
|
the final action when closing a file.<br />
|
|
<em>This field is present in version 0+ of the superblock.</em><br />
|
|
<em>The size of this field has been reduced from 4 bytes in superblock format versions 0 and 1 to
|
|
1 byte.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Base Address</td>
|
|
<td>This field is the same as described for versions 0 and 1 of the superblock.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Superblock Extension Address</td>
|
|
<td>The field is the address of the object header for the @ref subsec_fmt4_boot_supext. If there is no
|
|
extension then this entry should be the @ref FMT4UndefinedAddress "undefined address".</td>
|
|
</tr>
|
|
<tr>
|
|
<td>End of File Address</td>
|
|
<td>This field is the same as described for versions 0 and 1 of the superblock.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Root Group Object Header Address</td>
|
|
<td>This is the address of the @ref sec_fmt4_dataobject, which serves as the entry point into the group
|
|
graph for the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Superblock Checksum</td>
|
|
<td>The checksum for the superblock.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_boot_driver II.B. Disk Format: Level 0B - File Driver Info
|
|
The <strong>driver information block</strong> lives right after the superblock.
|
|
It is an optional region of the file which contains information
|
|
needed by the file driver in order to reopen a file and may only exist when superblock version is less than 2.
|
|
The format is described below:
|
|
<table>
|
|
<caption align=top><strong>Layout: Driver Information Block</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td colspan="3">Reserved</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Driver Information Size</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Driver Identification <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br /><br />Driver Information <em>(variable size)</em><br /><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align=top><strong>Fields: Driver Information Block</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>The version number of the Driver Information Block. This document describes version 0.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Driver Information Size</td>
|
|
<td>The size in bytes of the <em>Driver Information</em> field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Driver Identification</td>
|
|
<td>This is an eight-byte ASCII string without null termination which identifies the driver and/or version number
|
|
of the Driver Information block. The predefined driver encoded in this field by the HDF5 library is identified
|
|
by the letters <code>NCSA</code> followed by the first four characters of the driver name. If the Driver Information
|
|
Block is not the original version then the last letter(s) of the identification will be replaced by a version
|
|
number in ASCII, starting with 0.<br />
|
|
Identification for user-defined drivers is also eight-byte long. It can be arbitrary but should be unique to
|
|
avoid the four character prefix “NCSA”.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Driver Information</td>
|
|
<td>Driver information is stored in a format defined by the file driver (see description below).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The two drivers encoded in the <em>Driver Identification</em> field are as follows:
|
|
\li Multi driver:<br />The identifier for this driver is “NCSAmult”. This driver provides
|
|
a mechanism for segregating raw data and different types of metadata into multiple files. These files
|
|
are viewed by the library as a single virtual HDF5 file with a single file address. A maximum of 6
|
|
files will be created for the following data: superblock, B-tree, raw data, global heap, local heap,
|
|
and object header. More than one type of data can be written to the same file.
|
|
\li Family driver:<br />The identifier for this driver is “NCSAfami” and is encoded in this
|
|
field for library version 1.8 and after. This driver is designed for systems that do not support files
|
|
larger than 2 gigabytes by splitting the HDF5 file address space across several smaller files. It does
|
|
nothing to segregate metadata and raw data; they are mixed in the address space just as they would be
|
|
in a single contiguous file.
|
|
|
|
The format of the <em>Driver Information</em> field for the above two drivers are described below:
|
|
<table>
|
|
<caption><strong>Layout: Multi Driver Information</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Member Mapping</td>
|
|
<td>Member Mapping</td>
|
|
<td>Member Mapping</td>
|
|
<td>Member Mapping</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Member Mapping</td>
|
|
<td>Member Mapping</td>
|
|
<td>Reserved</td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of Member File 1<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />End of Address for Member File 1<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of Member File 2<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />End of Address for Member File 2<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />... ...<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of Member File N<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />End of Address for Member File N<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Name of Member File 1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Name of Member File 2 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />... ...<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Name of Member File N <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Multi Driver Information</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Member Mapping</td>
|
|
<td>These fields are integer values from 1 to 6 indicating how the data can be mapped to or
|
|
merged with another type of data.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Member Mapping</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">1</td>
|
|
<td>The superblock data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">2</td>
|
|
<td>The B-tree data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">3</td>
|
|
<td>The raw data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">4</td>
|
|
<td>The global heap data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">5</td>
|
|
<td>The local heap data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">6</td>
|
|
<td>The object header data.</td>
|
|
</tr>
|
|
</table><br />
|
|
For example, if the third field has the value 3 and all the rest have the
|
|
value 1, it means there are two files, one for raw data, and one for superblock,
|
|
B-tree, global heap, local heap, and object header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Reserved</td>
|
|
<td>These fields are reserved and should always be zero.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address of Member File N</td>
|
|
<td>This field specifies the virtual address at which the member file starts.<br />
|
|
N is the number of member files.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>End of Address for Member File N</td>
|
|
<td>This field is the end of allocated address for the member file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Name of Member File N</td>
|
|
<td>This field is the null-terminated name of member file. And its length should be multiples
|
|
of 8 bytes. Additional bytes will be padded with <em>NULL</em>s. The default naming convention is
|
|
<em>%%s-X.h5</em>, where <em>X</em> is one of the letters <em>s</em> (for superblock),
|
|
<em>b</em> (for B-tree), <em>r</em> (for raw data), <em>g</em> (for global heap),
|
|
<em>l</em> (for local heap), and <em>o</em> (for object header). The name for the whole
|
|
HDF5 file will substitute the <em>%s</em> in the string.</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Layout: Family Driver Information</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Member File<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Fields: Family Driver Information</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Member File</td>
|
|
<td>This field is the size of the member file in the family of files.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_boot_supext II.C. Disk Format: Level 0C - Superblock Extension
|
|
The <em>superblock extension</em> is used to store superblock metadata which is either optional, or added
|
|
after the version of the superblock was defined. Superblock extensions may only exist when version 2+ or
|
|
later of the superblock is used. A superblock extension is an object header which may hold the following messages:
|
|
\li \ref subsec_fmt4_infra_sohm containing information to locate the master table of shared object
|
|
header message indices.
|
|
\li \ref subsubsec_fmt4_dataobject_hdr_msg_btreek containing non-default B-tree ‘K’ values.
|
|
\li \ref subsubsec_fmt4_dataobject_hdr_msg_drvinfo containing information needed by the file driver in
|
|
order to reopen a file. See also the \ref subsec_fmt4_boot_driver section above.
|
|
\li \ref subsubsec_fmt4_dataobject_hdr_msg_fsinfo containing information about file space handling in the file.
|
|
\li \ref subsubsec_fmt4_dataobject_hdr_msg_mdci containing the location and size of a
|
|
@ref subsec_fmt4_infra_mdci "metadata cache image block", when one is present in the file.
|
|
|
|
\section sec_fmt4_infra III. Disk Format: Level 1 - File Infrastructure
|
|
|
|
\subsection subsec_fmt4_infra_btrees III.A. Disk Format: Level 1A - B-trees and B-tree Nodes
|
|
B-trees allow flexible storage for objects which tend to grow in ways that cause the object to be stored
|
|
discontiguously. B-trees are described in various algorithms books including "Introduction to Algorithms" by
|
|
Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest. The B-trees are used in several places in
|
|
the HDF5 file format, when an index is needed for another data structure.
|
|
|
|
The version 1 B-tree structure described below is the original index structure. The version 1 B-trees are
|
|
being phased out in favor of the version 2 B-trees described below. Note that both types of structures may
|
|
be found in the same file depending on the application settings when creating the file.
|
|
|
|
\subsubsection subsubsec_fmt4_infra_btrees_v1 III.A.1. Disk Format: Level 1A1 - Version 1 B-trees
|
|
Version 1 B-trees in HDF5 files an implementation of the B-tree. The sibling nodes at a
|
|
particular level in the tree are stored in a doubly-linked list. See the “Efficient Locking
|
|
for Concurrent Operations on B-trees” paper by Phillip Lehman and S. Bing Yao as published in the
|
|
<cite>ACM Transactions on Database Systems</cite>, Vol. 6, No. 4, December 1981.
|
|
|
|
The B-trees implemented by the file format contain one more key than the number of children. In other
|
|
words, each child pointer out of a B-tree node has a left key and a right key. The pointers out of internal
|
|
nodes point to sub-trees while the pointers out of leaf nodes point to symbol nodes and raw data chunks.
|
|
Aside from that difference, internal nodes and leaf nodes are identical.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: B-tree Nodes</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Node Type</td>
|
|
<td>Node Level</td>
|
|
<td colspan="2">Entries Used</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Address of Left Sibling<sup>O</sup><br /><br /></td>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Address of Right Sibling<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Key 1 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Address of Child 1<sup>O</sup><br /><br /></td>
|
|
<tr align="center">
|
|
<td colspan="4">Key 2 <em>(variable size)</em></td>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Address of Child 2<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Key 2<em>K</em> <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Address of Child 2<em>K</em><sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Key 2<em>K</em>+1 <em>(variable size)</em></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: B-tree Nodes</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>TREE</code>” is used to indicate the beginning of a
|
|
B-tree node. This gives file consistency checking utilities a better chance of reconstructing
|
|
a damaged file.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Node Type</td>
|
|
<td>Each B-tree points to a particular type of data. This field indicates the type of data as well as
|
|
implying the maximum degree <em>K</em> of the tree and the size of each Key field.<br />
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Node Type</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">0</td>
|
|
<td>This tree points to group nodes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">1</td>
|
|
<td>This tree points to a raw data chunk.</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Node Level</td>
|
|
<td>The node level indicates the level at which this node appears in the tree (leaf nodes are at level
|
|
zero). Not only does the level indicate whether child pointers point to sub-trees or to data, but it
|
|
can also be used to help file consistency checking utilities reconstruct damaged trees.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Entries Used</td>
|
|
<td>This determines the number of children to which this node points. All nodes of a particular type of
|
|
tree have the same maximum degree, but most nodes will point to less than that number of children. The
|
|
valid child pointers and keys appear at the beginning of the node and the unused pointers and keys
|
|
appear at the end of the node. The unused pointers and keys have undefined values.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Address of Left Sibling</td>
|
|
<td>This is the relative file address of the left sibling of the current node. If the current node is the
|
|
left-most node at this level then this field is the @ref FMT4UndefinedAddress "undefined address".</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Address of Right Sibling</td>
|
|
<td>This is the relative file address of the right sibling of the current node. If the current node is the
|
|
right-most node at this level then this field is the @ref FMT4UndefinedAddress "undefined address".</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Keys and Child Pointers</td>
|
|
<td>Each tree has 2<em>K</em>+1 keys with 2<em>K</em> child pointers interleaved between the keys. The number
|
|
of keys and child pointers actually containing valid values is determined by the node’s
|
|
<em>Entries Used</em> field. If that field is <em>N</em> then the B-tree contains <em>N</em> child
|
|
pointers and <em>N</em>+1 keys.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Key</td>
|
|
<td>The format and size of the key values is determined by the type of data to which this tree points. The
|
|
keys are ordered and are boundaries for the contents of the child pointer; that is, the key values
|
|
represented by child <em>N</em> fall between Key <em>N</em> and Key <em>N</em>+1. Whether the interval
|
|
is open or closed on each end is determined by the type of data to which the tree points.<br />
|
|
The format of the key depends on the node type. For nodes of node type 0 (group nodes), the key is
|
|
formatted as follows:
|
|
<table>
|
|
<tr valign=top align="left">
|
|
<td width=20%>A single field of <i>@ref FMT4SizeOfLengthsV0 "Size of Lengths"</i> bytes.</td>
|
|
<td>Indicates the byte offset into the local heap for the first object name in the subtree which
|
|
that key describes.</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
For nodes of node type 1 (chunked raw data nodes), the key is formatted as follows:
|
|
<table>
|
|
<tr valign=top align="left">
|
|
<td width=20%>Bytes 1-4</td>
|
|
<td>Size of chunk in bytes.</td>
|
|
</tr>
|
|
<tr valign=top align="left">
|
|
<td>Bytes 5-8</td>
|
|
<td>Filter mask, a 32-bit bit field indicating which filters have been skipped for this chunk. Each
|
|
filter has an index number in the pipeline (starting at 0, with the first filter to apply) and
|
|
if that filter is skipped, the bit corresponding to its index is set.</td>
|
|
</tr>
|
|
<tr valign=top align="left">
|
|
<td>(<em>d</em> + 1) 64-bit fields</td>
|
|
<td>The offset of the chunk within the dataset, where <b>d</b> is the number of dimensions
|
|
of the dataset. The first <em>d</em> values are the chunk’s offset along each real
|
|
dataset dimension. The extra (<em>d</em>+1)<sup>th</sup> value is not a real dimension
|
|
offset; it is a placeholder for the offset within the dataset’s datatype element and
|
|
is always <code>0</code>. For example, if a chunk in a 3-dimensional dataset begins at the
|
|
position <code>[5,5,5]</code>, there will be three such 64-bit indices, each with the value of
|
|
<code>5</code>, followed by a <code>0</code> value for the datatype-element placeholder.</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Child Pointer</td>
|
|
<td>The tree node contains file addresses of subtrees or data depending on the node level. Nodes at Level
|
|
0 point to data addresses, either raw data chunks or group nodes. Nodes at non-zero levels point to other
|
|
nodes of the same B-tree.<br />
|
|
For raw data chunk nodes, the child pointer is the address of a single raw data chunk. For group nodes,
|
|
the child pointer points to a @ref subsec_fmt4_infra_symboltableentry, which contains
|
|
information for multiple symbol table entries.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
Conceptually, each B-tree node looks like this:
|
|
<table>
|
|
<tr valign=top align="center">
|
|
<td>key[0]</td>
|
|
<td> </td>
|
|
<td>child[0]</td>
|
|
<td> </td>
|
|
<td>key[1]</td>
|
|
<td> </td>
|
|
<td>child[1]</td>
|
|
<td> </td>
|
|
<td>key[2]</td>
|
|
<td>; </td>
|
|
<td>...</td>
|
|
<td> </td>
|
|
<td>...</td>
|
|
<td> </td>
|
|
<td>key[<i>N</i>-1]</td>
|
|
<td> </td>
|
|
<td>child[<i>N</i>-1]</td>
|
|
<td> </td>
|
|
<td>key[<i>N</i>]</td>
|
|
</tr>
|
|
</table>
|
|
where child[<i>i</i>] is a pointer to a sub-tree (at a level above Level 0) or to data (at Level 0).
|
|
Each key[<i>i</i>] describes an <i>item</i> stored by the B-tree (a chunk or an object of a group node).
|
|
The range of values represented by child[<i>i</i>] is indicated by key[<i>i</i>] and key[<i>i</i>+1].
|
|
|
|
The following question must next be answered: “Is the value described by key[<i>i</i>] contained in
|
|
child[<i>i</i>-1] or in child[<i>i</i>]?” The answer depends on the type of tree. In trees for groups (node
|
|
type 0) the object described by key[<i>i</i>] is the greatest object contained in child[<i>i</i>-1] while
|
|
in chunk trees (node type 1) the chunk described by key[<i>i</i>] is the least chunk in child[<i>i</i>].
|
|
|
|
That means that key[0] for group trees is sometimes unused; it points to offset zero in the heap, which is
|
|
always the empty string and compares as "less-than" any valid object name.
|
|
|
|
And key[<i>N</i>] for chunk trees is sometimes unused; it contains a chunk offset which compares as
|
|
"greater-than" any other chunk offset and has a chunk byte size of zero to indicate that it is not actually
|
|
allocated.
|
|
|
|
\subsubsection subsubsec_fmt4_infra_btrees_v2 III.A.2. Disk Format: Level 1A2 - Version 2 B-trees
|
|
Version 2 (v2) B-trees are “traditional” B-trees, with one major difference. Instead of just using
|
|
a simple pointer (or address in the file) to a child of an internal node, the pointer to the child node
|
|
contains two additional pieces of information: the number of records in the child node itself, and the
|
|
total number of records in the child node and all its descendants. Storing this additional information
|
|
allows fast array-like indexing to locate the n<sup>th</sup> record in the B-tree.
|
|
|
|
The entry into a version 2 B-tree is a header which contains global information about the structure of
|
|
the B-tree. The <em>root node address</em> field in the header points to the B-tree root node, which is
|
|
either an internal or leaf node, depending on the value in the header’s <em>depth</em> field. An
|
|
internal node consists of records plus pointers to further leaf or internal nodes in the tree. A leaf
|
|
node consists of solely of records. The format of the records depends on the B-tree type (stored in
|
|
the header).
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree Header</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Type</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Node Size</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Record Size</td>
|
|
<td colspan="2">Depth</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Split Percent</td>
|
|
<td>Merge Percent</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Root Node Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Number of Records in Root Node</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Total Number of Records in B-tree<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree Header</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>BTHD</code>” is used to indicate the header of a
|
|
version 2 (v2) B-tree node.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this B-tree header. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Type</td>
|
|
<td>This field indicates the type of B-tree:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">0</td>
|
|
<td>This B-tree is used for testing only. This value should <em>not</em> be used for storing
|
|
records in actual HDF5 files.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">1</td>
|
|
<td>This B-tree is used for indexing indirectly accessed, non-filtered ‘huge’
|
|
fractal heap objects.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">2</td>
|
|
<td>This B-tree is used for indexing indirectly accessed, filtered ‘huge’
|
|
fractal heap objects.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">3</td>
|
|
<td>This B-tree is used for indexing directly accessed, non-filtered ‘huge’
|
|
fractal heap objects.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">4</td>
|
|
<td>This B-tree is used for indexing directly accessed, filtered ‘huge’
|
|
fractal heap objects.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">5</td>
|
|
<td>This B-tree is used for indexing the ‘name’ field for links in indexed
|
|
groups.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">6</td>
|
|
<td>This B-tree is used for indexing the ‘creation order’ field for links
|
|
in indexed groups.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">7</td>
|
|
<td>This B-tree is used for indexing shared object header messages.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">8</td>
|
|
<td>This B-tree is used for indexing the ‘name’ field for indexed
|
|
attributes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">9</td>
|
|
<td>This B-tree is used for indexing the ‘creation order’ field for
|
|
indexed attributes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">10</td>
|
|
<td>This B-tree is used for indexing chunks of datasets with no filters and with more
|
|
than one dimension of unlimited extent.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">11</td>
|
|
<td>This B-tree is used for indexing chunks of datasets with filters and more than one
|
|
dimension of unlimited extent.</td>
|
|
</tr>
|
|
</table>
|
|
The format of records for each type is described below.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Node Size</td>
|
|
<td>This is the size in bytes of all B-tree nodes.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Record Size</td>
|
|
<td>This field is the size in bytes of the B-tree record.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Depth</td>
|
|
<td>This is the depth of the B-tree.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Split Percent</td>
|
|
<td>The percent full that a node needs to increase above before it is split.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Merge Percent</td>
|
|
<td>The percent full that a node needs to be decrease below before it is split.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Root Node Address</td>
|
|
<td>This is the address of the root B-tree node. A B-tree with no records will have the
|
|
@ref FMT4UndefinedAddress "undefined address" in this field.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Number of Records in Root Node</td>
|
|
<td>This is the number of records in the root node.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Total Number of Records in B-tree</td>
|
|
<td>This is the total number of records in the entire B-tree.</td>
|
|
</tr>
|
|
<tr valign="top">
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the B-tree header.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree Internal Node</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Type</td>
|
|
<td colspan="2">Record \#0, Record \#1, ... Record \#(R-1) <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Node Pointer \#0<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Records for Child Node \#0 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Total Number of Records for Child Node \#0 <em>(optional, variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Node Pointer \#1<sup>O</sup><br /> <br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Records for Child Node \#1 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Total Number of Records for Child Node \#1 <em>(optional, variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Node Pointer \#R<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Records for Child Node \#R <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Total Number of Records for Child Node \#R <em>(optional, variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree Internal Node</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “ <code>BTIN</code> ” is used to indicate the internal node
|
|
of a B-tree.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this B-tree internal node. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Type</td>
|
|
<td>This field is the type of the B-tree node. It should always be the same as the B-tree type in
|
|
the header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Record \#i (i = 0 to (R-1))</td>
|
|
<td>Let <em>R</em> be the number of records for this node: for the root node, <em>R</em> is given
|
|
by the B-tree Header's <em>Number of Records in Root Node</em> field; for any other node,
|
|
<em>R</em> is given by the <em>Number of Records for Child Node \#j</em> field in this node's
|
|
parent, specifically the pointer triplet whose <em>Child Node Pointer \#j</em> gave this
|
|
node's address. The size of this field is
|
|
determined by <em>R</em> and the <em>Record Size</em> (the latter always from the header); the
|
|
format of records depends on the type of B-tree. Because each internal node has one more child
|
|
pointer triplet than it has records, this node has <em>R</em>+1 child node pointers, indexed
|
|
\#0 through \#R (see below).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Child Node Pointer \#j (j = 0 to R)</td>
|
|
<td>This field is the address of child \#j pointed to by this internal node. Together with the
|
|
following field, <em>Number of Records for Child Node \#j</em>, and (where present) the field
|
|
after that, <em>Total Number of Records for Child Node \#j</em>, this forms what the rest of
|
|
this table calls a <em>pointer triplet</em> — the repeating group of per-child fields
|
|
between <em>Child Node Pointer \#j</em> and <em>Child Node Pointer \#(j+1)</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Records for Child Node \#j (j = 0 to R)</td>
|
|
<td>This is the number of records in child \#j (see the <em>Child Node Pointer</em> field above).<br />
|
|
The number of bytes used to store this field is determined by the maximum possible number of records able
|
|
to be stored in the child node.<br />
|
|
The maximum number of records in a child node is computed in the following way:
|
|
<ul>
|
|
<li>Subtract the fixed size overhead for the child node (for example, its signature, version,
|
|
checksum, and so on, and one pointer triplet of information for the child node, per the
|
|
R+1 child pointers noted above) from the <em>Node Size</em> (see the header).</li>
|
|
<li>Divide that result by the sum of the <em>Record Size</em> and the pointer triplet of information
|
|
stored to reach each child node from this node.</li>
|
|
</ul><br />
|
|
Note that leaf nodes do not encode any child pointer triplets, so the maximum number of records in a
|
|
leaf node is just the <em>Node Size</em> minus the leaf node overhead, divided by the <em>Record Size</em>.<br />
|
|
Also note that the first level of internal nodes above the leaf nodes — the “twig”
|
|
internal nodes — do not encode a <em>Total Number of Records for Child Node \#j</em> value
|
|
(since it would be the same as <em>Number of Records for Child Node \#j</em>), so the maximum number
|
|
of records in these nodes is computed with the equation above, but using (<em>Child Node Pointer
|
|
\#j</em>, <em>Number of Records for Child Node \#j</em>) pairs instead of full triplets.<br />
|
|
The number of bytes used to encode this field is the least number of bytes required to encode the
|
|
maximum number of records in a child node value for the child nodes below this level in the B-tree.<br />
|
|
For example, if the maximum number of child records is 123, one byte will be used to encode these
|
|
values in this node; if the maximum number of child records is 20000, two bytes will be used to
|
|
encode these values in this node; and so on. The maximum number of bytes used to encode these values
|
|
is 8 (in other words, an unsigned 64-bit integer).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Total Number of Records for Child Node \#j (j = 0 to R)</td>
|
|
<td>This is the total number of records for child \#j and all its descendants. This field exists
|
|
only in nodes whose depth in the B-tree node is greater than 1; see the note about
|
|
“twig” internal nodes under <em>Number of Records for Child Node \#j</em> above.<br />
|
|
The number of bytes used to store this field is determined by the maximum possible number of records
|
|
able to be stored in the child node and its descendants.<br />
|
|
The maximum possible number of records able to be stored in a child node and its descendants is
|
|
computed iteratively, in the following way: The maximum number of records in a leaf node is
|
|
computed, then that value is used to compute the maximum possible number of records in the first
|
|
level of internal nodes above the leaf nodes. Multiplying these two values together determines the
|
|
maximum possible number of records in child node pointers for the level of nodes two levels above
|
|
leaf nodes. This process is continued up to any level in the B-tree.<br />
|
|
The number of bytes used to encode this value is computed in the same way as for the <em>Number
|
|
of Records for Child Node \#j</em> field.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for this node.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree Leaf Node</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Type</td>
|
|
<td colspan="2">Record \#0, Record \#1, ... Record \#(R-1) <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree Leaf Node</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “ <code>BTLF</code> “ is used to indicate the leaf node
|
|
of a version 2 (v2) B-tree.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this B-tree leaf node. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Type</td>
|
|
<td>This field is the type of the B-tree node. It should always be the same as the B-tree type in
|
|
the header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Record \#i (i = 0 to (R-1))</td>
|
|
<td>Let <em>R</em> be the number of records for this node: for the root node, <em>R</em> is given
|
|
by the B-tree Header's <em>Number of Records in Root Node</em> field; for any other node,
|
|
<em>R</em> is given by the <em>Number of Records for Child Node \#j</em> field in this leaf's
|
|
parent (always a “twig” internal node, so this is one of a pointer pair, not a full
|
|
triplet). The size of this field is determined by <em>R</em> and the <em>Record Size</em> (the
|
|
latter always from the header); the format of records depends on the type of B-tree.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for this node.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The record layout for each stored (in other words, non-testing) B-tree type is as follows:
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 1 Record Layout - Indirectly Accessed, Non-Filtered,
|
|
‘Huge’ Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Huge Object Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Huge Object Length<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Huge Object ID<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 1 Record Layout - Indirectly Accessed, Non-Filtered,
|
|
‘Huge’ Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Huge Object Address</td>
|
|
<td>The address of the huge object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Huge Object Length</td>
|
|
<td>The length of the huge object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Huge Object ID</td>
|
|
<td>The heap ID for the huge object.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 2 Record Layout - Indirectly Accessed, Filtered, ‘Huge’
|
|
Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Filtered Huge Object Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Filtered Huge Object Length<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Filtered Huge Object Memory Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Huge Object ID<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 2 Record Layout - Indirectly Accessed, Filtered, ‘Huge’
|
|
Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Filtered Huge Object Address</td>
|
|
<td>The address of the filtered huge object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filtered Huge Object Length</td>
|
|
<td>The length of the filtered huge object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filter Mask</td>
|
|
<td>A 32-bit bit field indicating which filters have been skipped for this chunk. Each filter has
|
|
an index number in the pipeline (starting at 0, with the first filter to apply) and if that filter
|
|
is skipped, the bit corresponding to its index is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filtered Huge Object Memory Size</td>
|
|
<td>The size of the de-filtered huge object in memory.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Huge Object ID</td>
|
|
<td>The heap ID for the huge object.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 3 Record Layout - Directly Accessed, Non-Filtered, ‘Huge’
|
|
Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Huge Object Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Huge Object Length<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 3 Record Layout - Directly Accessed, Non-Filtered, ‘Huge’
|
|
Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Huge Object Address</td>
|
|
<td>The address of the huge object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Huge Object Length</td>
|
|
<td>The length of the huge object in the file.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 4 Record Layout - Directly Accessed, Filtered, ‘Huge’
|
|
Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Filtered Huge Object Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Filtered Huge Object Length<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Filtered Huge Object Memory Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 4 Record Layout - Directly Accessed, Filtered, ‘Huge’
|
|
Fractal Heap Objects</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Filtered Huge Object Address</td>
|
|
<td>The address of the filtered huge object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filtered Huge Object Length</td>
|
|
<td>The length of the filtered huge object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filter Mask</td>
|
|
<td>A 32-bit bit field indicating which filters have been skipped for this chunk. Each filter has an
|
|
index number in the pipeline (starting at 0, with the first filter to apply) and if that filter
|
|
is skipped, the bit corresponding to its index is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filtered Huge Object Memory Size</td>
|
|
<td>The size of the de-filtered huge object in memory.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 5 Record Layout - Link Name for Indexed Group</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Hash of Name</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">ID <em>(bytes 1-4)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="3">ID <em>(bytes 5-7)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 5 Record Layout - Link Name for Indexed Group</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Hash</td>
|
|
<td>This field is hash value of the name for the link. The hash value is the Jenkins’ lookup3
|
|
checksum algorithm applied to the link’s name.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>ID</td>
|
|
<td>This is a 7-byte sequence of bytes and is the heap ID for the link record in the group’s
|
|
fractal heap.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 6 Record Layout - Creation Order for Indexed Group</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td colspan="4"><br />Creation Order <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">ID <em>(bytes 1-4)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="3">ID <em>(bytes 5-7)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 6 Record Layout - Creation Order for Indexed Group</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Creation Order</td>
|
|
<td>This field is the creation order value for the link.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>ID</td>
|
|
<td>This is a 7-byte sequence of bytes and is the heap ID for the link record in the group’s
|
|
fractal heap.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 7 Record Layout - Shared Object Header Messages
|
|
(Sub-Type 0 - Message in Heap)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan>Message Location</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Hash</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Reference Count</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Heap ID <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 7 Record Layout - Shared Object Header Messages
|
|
(Sub-Type 0 - Message in Heap)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Location</td>
|
|
<td>This field Indicates the location where the message is stored:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">0</td>
|
|
<td>Shared message is stored in shared message index heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">1</td>
|
|
<td>Shared message is stored in object header.</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Hash</td>
|
|
<td>This field is hash value of the shared message. The hash value is the Jenkins’ lookup3
|
|
checksum algorithm applied to the shared message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Reference Count</td>
|
|
<td>The number of objects which reference this message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Heap ID</td>
|
|
<td>This is an 8-byte sequence of bytes and is the heap ID for the shared message in the shared
|
|
message index’s fractal heap.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 7 Record Layout - Shared Object Header Messages
|
|
(Sub-Type 1 - Message in Object Header)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan>Message Location</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Hash</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Reserved (zero)</td>
|
|
<td>Message Type</td>
|
|
<td colspan="2">Object Header Index</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Object Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 7 Record Layout - Shared Object Header Messages
|
|
(Sub-Type 1 - Message in Object Header)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Location</td>
|
|
<td>This field Indicates the location where the message is stored:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">0</td>
|
|
<td>Shared message is stored in shared message index heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">1</td>
|
|
<td>Shared message is stored in object header.</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Hash</td>
|
|
<td>This field is hash value of the shared message. The hash value is the Jenkins’ lookup3
|
|
checksum algorithm applied to the shared message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Type</td>
|
|
<td>The object header message type of the shared message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Object Header Index</td>
|
|
<td>This field indicates that the shared message is the n<sup>th</sup> message of its type in the
|
|
specified object header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Object Header Address</td>
|
|
<td>The address of the object header containing the shared message.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 8 Record Layout - Attribute Name for Indexed Attributes</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Heap ID <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan>Message Flags</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Creation Order</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Hash of Name</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 8 Record Layout - Attribute Name for Indexed Attributes</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Heap ID</td>
|
|
<td>This is an 8-byte sequence of bytes and is the heap ID for the attribute in the object’s
|
|
attribute fractal heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Flags</td>
|
|
<td>The object header message flags for the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Creation Order</td>
|
|
<td>This field is the creation order value for the attribute.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Hash</td>
|
|
<td>This field is hash value of the name for the attribute. The hash value is the Jenkins’
|
|
lookup3 checksum algorithm applied to the attribute’s name.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree, Type 9 Record Layout- Creation Order for Indexed Attributes</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Heap ID <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan>Message Flags</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Creation Order</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 9 Record Layout- Creation Order for Indexed Attributes</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Heap ID</td>
|
|
<td>This is an 8-byte sequence of bytes and is the heap ID for the attribute in the object’s
|
|
attribute fractal heap.</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>Message Flags</td>
|
|
<td>The object header message flags for the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Creation Order</td>
|
|
<td>This field is the creation order value for the attribute.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption>\anchor FMT4V2BtType10 <strong>Layout: Version 2 B-tree, Type 10 Record Layout - Non-filtered Dataset Chunks</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#0 Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#1 Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />...<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#(d-1) Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li In the above table, <b>d</b> is the number of dimensions in the dataset’s dataspace. Unlike
|
|
the version 1 B-tree chunk key, there is no extra field for the dataset’s datatype element;
|
|
exactly <em>d</em> scaled offsets are stored, one per real dataset dimension.
|
|
|
|
<table>
|
|
<caption>\anchor FMT4V2BtType10Fields <strong>Fields: Version 2 B-tree, Type 10 Record Layout - Non-filtered Dataset Chunks</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This field is the address of the dataset chunk in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension \#i Scaled Offset (i = 0 to (d-1))</td>
|
|
<td>This field is the scaled offset of the chunk within the dataset, where <b>d</b> is the number of
|
|
dimensions for the dataset. The first scaled offset stored in the list is for the slowest
|
|
changing dimension, and the last scaled offset stored is for the fastest changing dimension.
|
|
Scaled offset is calculated by dividing the chunk dimension sizes into the chunk offsets.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption>\anchor FMT4V2BtType11 <strong>Layout: Version 2 B-tree, Type 11 Record Layout - Filtered Dataset Chunks (Layout Version 4)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Chunk Size <em>(variable size; at most 8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#0 Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#1 Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />...<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#(d-1) Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li See the note for @ref FMT4V2BtType10 "Type 10 Record Layout" above for the definition of <b>d</b> and the scaled offset fields.
|
|
\li This encoding is used when this B-Tree is used as an index for a dataset with layout message version 4.
|
|
|
|
<table>
|
|
<caption>\anchor FMT4V2BtType11V5 <strong>Layout: Version 2 B-tree, Type 11 Record Layout - Filtered Dataset Chunks (Layout Version 5)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Chunk Size<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#0 Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#1 Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />...<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#(d-1) Scaled Offset <em>(8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li See the note for @ref FMT4V2BtType10 "Type 10 Record Layout" above for the definition of <b>d</b> and the scaled offset fields.
|
|
\li This encoding is used when this B-tree is used as an index for a dataset with layout message version 5 or above.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree, Type 11 Record Layout - Filtered Dataset Chunks</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This field is the address of the dataset chunk in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Chunk Size</td>
|
|
<td>This field is the size of the dataset chunk in bytes. The length of this field is determined depending on the version of the associated layout message as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Layout Message Version</th>
|
|
<th align="left">Chunk Size Encoding Length</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>One more than the number of bytes that would be needed to encode the size of an unfiltered chunk, but not more than 8.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code> or above</td>
|
|
<td>The size specified in the “@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filter Mask</td>
|
|
<td>This field is the filter mask which indicates the filter
|
|
to skip for the dataset chunk. Each filter has an index
|
|
number in the pipeline and if that filter is skipped,
|
|
the bit corresponding to its index is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension \#i Scaled Offset (i = 0 to (d-1))</td>
|
|
<td>See the @ref FMT4V2BtType10Fields "Dimension #i Scaled Offset" field description in the Type 10 Record Layout Fields table above.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_infra_symboltable III.B. Disk Format: Level 1B - Group Symbol Table Nodes
|
|
A group is an object internal to the file that allows arbitrary nesting of objects within the file (including
|
|
other groups). A group maps a set of link names in the group to a set of relative file addresses of objects
|
|
in the file. Certain metadata for an object to which the group points can be cached in
|
|
object’s header.
|
|
|
|
An HDF5 object name space can be stored hierarchically by partitioning the name into components and storing
|
|
each component as a link in a group. The link for a non-ultimate component points to the group containing the
|
|
next component. The link for the last component points to the object being named.
|
|
|
|
One implementation a group is a collection of symbol table nodes indexed by a B-tree. Each symbol table
|
|
node contains entries for one or more links. If an attempt is made to add a link to an already full
|
|
symbol table node containing 2<em>K</em> entries, then the node is split and one node contains <em>K</em>
|
|
symbols and the other contains <em>K</em>+1 symbols.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Symbol Table Node (A Leaf of a B-tree)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version Number</td>
|
|
<td>Reserved <em>(zero)</em></td>
|
|
<td colspan="2">Number of Symbols</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br /><br />Group Entries<br /><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Symbol Table Node (A Leaf of a B-tree)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string <code>SNOD</code> is used to indicate the beginning of a symbol table node. This
|
|
gives file consistency checking utilities a better chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version Number</td>
|
|
<td>The version number for the symbol table node. This document describes version 1. (There is no version
|
|
‘0’ of the symbol table node)</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Number of Symbols</td>
|
|
<td>Although all symbol table nodes have the same length, most contain fewer than the maximum possible number of
|
|
link entries. This field indicates how many entries contain valid data. The valid entries are packed
|
|
at the beginning of the symbol table node while the remaining entries contain undefined values.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Group Entries</td>
|
|
<td>Each link has an entry in the symbol table node. The format of the entry is described below. There are
|
|
2<em>K</em> entries in each group node, where <em>K</em> is the “Group Leaf Node K” value
|
|
from the @ref subsec_fmt4_boot_super.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_infra_symboltableentry III.C. Disk Format: Level 1C - Symbol Table Entry
|
|
Each symbol table entry in a symbol table node is designed to allow for very fast browsing of stored objects.
|
|
Toward that design goal, the symbol table entries include space for caching certain constant metadata from the
|
|
object header.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Symbol Table Entry</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Link Name Offset<sup>L</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Object Header Address<sup>O</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Cache Type</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br /><br />Scratch-pad Space <em>(16 bytes)</em><br /><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Symbol Table Entry</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Link Name Offset</td>
|
|
<td>This is the byte offset into the group’s local heap for the name of the link. The name is null
|
|
terminated.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Object Header Address</td>
|
|
<td>Every object has an object header which serves as a permanent location for the object’s metadata.
|
|
In addition to appearing in the object header, some of the object’s metadata can be cached in the
|
|
scratch-pad space.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Cache Type</td>
|
|
<td>The cache type is determined from the object header. It also determines the format for the scratch-pad
|
|
space.<br />
|
|
<table>
|
|
<tr align="left">
|
|
<th>Type:</th>
|
|
<th>Description:</th>
|
|
</tr>
|
|
<tr>
|
|
<td width="10%" align="center">0</td>
|
|
<td>No data is cached by the group entry. This is guaranteed to be the case when an object header has
|
|
a link count greater than one.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">1</td>
|
|
<td>Group object header metadata is cached in the scratch-pad space. This implies that the symbol table
|
|
entry refers to another group.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">2</td>
|
|
<td>The entry is a symbolic link. The first four bytes of the scratch-pad space are the offset into
|
|
the local heap for the link value. The object header address will be undefined.</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Reserved</td>
|
|
<td>These four bytes are present so that the scratch-pad space is aligned on an eight-byte boundary. They
|
|
are always set to zero.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Scratch-pad Space</td>
|
|
<td>This space is used for different purposes, depending on the value of the Cache Type field. Any meta-data
|
|
about an object represented in the scratch-pad space is duplicated in the object header for
|
|
that object.<br />
|
|
Furthermore, no data is cached in the group entry scratch-pad space if the object header for the object
|
|
has a link count greater than one.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_infra_symboltableentry_scratch Format of the Scratch-pad Space
|
|
The symbol table entry scratch-pad space is formatted according to the value in the Cache Type field.
|
|
|
|
If the Cache Type field contains the value zero (<code>(0)</code>) then no information is stored in the
|
|
scratch-pad space.
|
|
|
|
If the Cache Type field contains the value one (<code>1</code>), then the scratch-pad space contains
|
|
cached metadata for another object header in the following format:
|
|
<table>
|
|
<caption align="top"><strong>Layout: Object Header Scratch-pad Format</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Address of B-tree<sup>O</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Address of Name Heap<sup>O</sup></td>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Object Header Scratch-pad Format</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Address of B-tree</td>
|
|
<td>This is the file address for the root of the group’s B-tree.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Address of Name Heap</td>
|
|
<td>This is the file address for the group’s local heap, in which are stored the group’s
|
|
symbol names.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
If the Cache Type field contains the value two (<code>(2)</code>), then the scratch-pad space contains
|
|
cached metadata for a symbolic link in the following format:
|
|
<table>
|
|
<caption align="top"><strong>Layout: Symbolic Link Scratch-pad Format</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Offset to Link Value</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Symbolic Link Scratch-pad Format</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Offset to Link Value</td>
|
|
<td>The value of a symbolic link (that is, the name of the thing to which it points) is stored in the
|
|
local heap. This field is the 4-byte offset into the local heap for the start of the link value, which
|
|
is null terminated.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_infra_localheap III.D. Disk Format: Level 1D - Local Heaps
|
|
A local heap is a collection of small pieces of data that are particular to a single object in the HDF5 file.
|
|
Objects can be inserted and removed from the heap at any time. The address of a heap does not change once
|
|
the heap is created. For example, a group stores addresses of objects in symbol table nodes with the names
|
|
of links stored in the group’s local heap.
|
|
<table>
|
|
<caption align="top"><strong>Layout: Local Heap</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td colspan="3">Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Data Segment Size<sup>L</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Offset to Head of Free-list<sup>L</sup></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Address of Data Segment<sup>O</sup></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Local Heap</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>HEAP</code> ” is used to indicate the beginning of a heap.
|
|
This gives file consistency checking utilities a better chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Each local heap has its own version number so that new heaps can be added to old files. This document
|
|
describes version zero (0) of the local heap.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Data Segment Size</td>
|
|
<td>The total amount of disk memory allocated for the heap data. This may be larger than the amount of space
|
|
required by the objects stored in the heap. The extra unused space in the heap holds a linked list of
|
|
free blocks.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Offset to Head of Free-list</td>
|
|
<td>This is the offset within the heap data segment of the first free block (or the
|
|
@ref FMT4UndefinedAddress "undefined address" if there is no no free block). The free block
|
|
contains “@ref FMT4SizeOfLengthsV0 "Size of Lengths"” bytes that are the offset of the next free block (or the value
|
|
‘1’ if this is the last free block) followed by “@ref FMT4SizeOfLengthsV0 "Size of Lengths"” bytes that
|
|
store the size of this free block. The size of the free block includes the space used to store the
|
|
offset of the next free block and the size of the current block, making the minimum size of a free
|
|
block 2 * “@ref FMT4SizeOfLengthsV0 "Size of Lengths"”.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Address of Data Segment</td>
|
|
<td>The data segment originally starts immediately after the heap header, but if the data segment must grow
|
|
as a result of adding more objects, then the data segment may be relocated, in its entirety, to another
|
|
part of the file.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
Objects within a local heap should be aligned on an 8-byte boundary.
|
|
|
|
\subsection subsec_fmt4_infra_globalheap III.E. Disk Format: Level 1E - Global Heap
|
|
Each HDF5 file has a global heap which stores various types of information which is typically shared between
|
|
datasets. The global heap was designed to satisfy these goals:
|
|
<ol type="A">
|
|
<li>Repeated access to a heap object must be efficient without resulting in repeated file I/O requests.
|
|
Since global heap objects will typically be shared among several datasets, it is probable that the
|
|
object will be accessed repeatedly.</li>
|
|
<li>Collections of related global heap objects should result in fewer and larger I/O requests. For
|
|
instance, a dataset of object references will have a global heap object for each reference. Reading
|
|
the entire set of object references should result in a few large I/O requests instead of one small
|
|
I/O request for each reference.</li>
|
|
<li>It should be possible to remove objects from the global heap and the resulting file hole should be
|
|
eligible to be reclaimed for other uses.</li>
|
|
</ol>
|
|
|
|
The implementation of the heap makes use of the memory management already available at the file level and
|
|
combines that with a new object called a <em>collection</em> to achieve goal B. The global heap is
|
|
the set of all collections. Each global heap object belongs to exactly one collection and each collection
|
|
contains one or more global heap objects. For the purposes of disk I/O and caching, a collection is treated
|
|
as an atomic object, addressing goal A.
|
|
|
|
When a global heap object is deleted from a collection (which occurs when its reference count falls to zero),
|
|
objects located after the deleted object in the collection are packed down toward the beginning of the
|
|
collection and the collection’s global heap object \#0 is created (if possible) or its size is increased
|
|
to account for the recently freed space. There are no gaps between objects in each collection, with the possible
|
|
exception of the final space in the collection, if it is not large enough to hold the header for the
|
|
collection’s global heap object \#0. These features address goal C.
|
|
|
|
The HDF5 library creates global heap collections as needed, so there may be multiple collections throughout
|
|
the file. The set of all of them is abstractly called the “global heap”, although they do not
|
|
actually link to each other, and there is no global place in the file where you can discover all of the
|
|
collections. The collections are found simply by finding a reference to one through another object in the file.
|
|
For example, data of variable-length datatype elements is stored in the global heap and is accessed via a
|
|
global heap ID. The format for global heap IDs is described at the end of this section.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: A Global Heap Collection</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td colspan="3">Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Collection Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Global Heap Object \#1<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Global Heap Object \#2<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />...<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Global Heap Object \#N<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Global Heap Object \#0 <em>(free space)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: A Global Heap Collection</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>GCOL</code>” is used to indicate the beginning of a collection.
|
|
This gives file consistency checking utilities a better chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>Each collection has its own version number so that new collections can be added to old files. This
|
|
document describes version one (1) of the collections (there is no version zero (0)).</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Collection Size</td>
|
|
<td>This is the size in bytes of the entire collection including this field. The default (and minimum)
|
|
collection size is 4096 bytes which is a typical file system block size. This allows for 127 16-byte
|
|
heap objects plus their overhead (the collection header of 16 bytes and the 16 bytes of information
|
|
about each heap object).</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Global Heap Object \#1 through \#N</td>
|
|
<td>Let <em>N</em> be the number of real (non-free-space) objects in this collection. <em>N</em> is not
|
|
stored explicitly; a reader determines it by parsing objects sequentially, in the order stored, until
|
|
either the free-space object (Global Heap Object \#0, Heap Object Index 0) is reached or the
|
|
Collection Size is exhausted. The objects are stored in any order with no intervening unused space.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Global Heap Object \#0</td>
|
|
<td>Global Heap Object \#0 (zero), when present, represents the free space in the collection. Free space
|
|
always appears at the end of the collection. If the free space is too small to store the header for
|
|
Object \#0 (described below) then the header is implied and is not written.<br />
|
|
The field <em>Object Size</em> for Object \#0 indicates the amount of possible free space in the
|
|
collection including the 16-byte header size of Object \#0.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Global Heap Object</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Heap Object Index</td>
|
|
<td colspan="2">Reference Count</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Object Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Object Data<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Global Heap Object</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Heap Object Index</td>
|
|
<td>Each object has a unique identification number within a collection. The identification numbers are
|
|
chosen so that new objects have the smallest value possible with the exception that the identifier
|
|
<code>0</code> always refers to the object which represents all free space within the collection.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Reference Count</td>
|
|
<td>All heap objects have a reference count field. An object which is referenced from some other part of the
|
|
file will have a positive reference count. The reference count for Object \#0 is always zero.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Reserved</td>
|
|
<td>Zero padding to align next field on an 8-byte boundary.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Object Size</td> <td>This is the size of the object data stored for the object. The actual storage space
|
|
allocated for the object data is rounded up to a multiple of eight.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Object Data</td>
|
|
<td>The object data is treated as a one-dimensional array of bytes to be interpreted by the caller.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<br />
|
|
\anchor FMT4GlobalHeapID <h4>The format for the ID used to locate an object in the global heap is described here:</h4>
|
|
<table>
|
|
<caption><strong>Layout: Global Heap ID</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Collection Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Object Index</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Global Heap ID</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Collection Address</td>
|
|
<td>This field is the address of the global heap collection where the data object is stored.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>ID</td>
|
|
<td>This field is the index of the data object within the global heap collection.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\note When used in @ref subsec_fmt4_dataobject_storage "Data Object Data Storage",
|
|
each Global Heap ID is preceded by additional fields not shown in the layout above.
|
|
For variable-length data, a 4-byte unsigned integer sequence length precedes each Global Heap ID.
|
|
For reference data, a 2-byte encoding header and a 4-byte unsigned integer encoded size
|
|
precede each Global Heap ID.
|
|
|
|
\subsection subsec_fmt4_infra_globalheapvds III.F. Disk Format: Level 1F - Global Heap Block for Virtual Datasets
|
|
The layout for the global heap block used with virtual datasets is described below. For more information
|
|
on global heaps, see “ @ref subsec_fmt4_infra_globalheap ”
|
|
|
|
There are two versions of the Virtual Dataset Global Heap Block format:
|
|
|
|
\li Version 0 is the original format without shared string support.
|
|
\li Version 1 introduces a new flags field that supports shared filenames and dataset names to reduce
|
|
storage requirements when the same names are used across multiple mappings. Version 1 will only
|
|
be used when the lower bound for the HDF5 library version bounds is set to 2.0 or later.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Global Heap Block for Virtual Dataset (Version 0)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Num Entries<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Filename \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Dataset \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Selection \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Virtual Selection \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Filename \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Dataset \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Selection \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Virtual Selection \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<br />
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Global Heap Block for Virtual Dataset (Version 1)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Num Entries<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Flags \#1</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Filename \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Dataset \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Selection \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Virtual Selection \#1 <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Flags \#n</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Filename \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Dataset \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Source Selection \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Virtual Selection \#n <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Global Heap Block for Virtual Dataset</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for the block; the value is 0 for version 0 format, 1 for version 1 format.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Num Entries<sup>L</sup></td>
|
|
<td>The number of entries in the block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags \#n</td>
|
|
<td>This field is a bit field containing additional information about the mapping.
|
|
<em>This field is present only in version 1 of the format.</em>
|
|
<table>
|
|
<tr>
|
|
<th width="15%">Bit</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>0</td>
|
|
<td>If set, the source filename is shared</td>
|
|
</tr>
|
|
<tr>
|
|
<td>1</td>
|
|
<td>If set, the source dataset name is shared</td>
|
|
</tr>
|
|
<tr>
|
|
<td>2</td>
|
|
<td>If set, the source file is the same as the virtual file</td>
|
|
</tr>
|
|
<tr>
|
|
<td>3-7</td>
|
|
<td>Reserved for future use</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Source Filename \#n <em>(variable size)</em></td>
|
|
<td>The source file name where the source dataset is located.<br /><br />
|
|
If the source filename is shared (bit 0 of flags is set), this is a "Size of Lengths"
|
|
length unsigned integer containing the index into the array of mappings where the source
|
|
filename is stored. For example, if the value stored here is 1, then the actual string
|
|
value for this field is stored in the Source Filename 1 field. If the source filename
|
|
is not shared, this is stored as a NULL terminated string. If the source file is the
|
|
same as the virtual file (bit 2 of flags is set), this field is not present.<br /><br />
|
|
<em>For version 0 format, this is always stored as a NULL terminated string.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Source Dataset \#n <em>(variable size)</em></td>
|
|
<td>The source dataset name that is mapped to the virtual dataset.<br /><br />
|
|
If the source dataset name is shared (bit 1 of flags is set), this is a "Size of Lengths"
|
|
length unsigned integer containing the index into the array of mappings where the source
|
|
dataset name is stored. For example, if the value stored here is 1, then the actual string
|
|
value for this field is stored in the Source Dataset 1 field. If the source dataset name
|
|
is not shared, this is stored as a NULL terminated string.<br /><br />
|
|
<em>For version 0 format, this is always stored as a NULL terminated string.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Source Selection \#n <em>(variable size)</em></td>
|
|
<td>The @ref FMT4DataspaceSEL "dataspace selection" in the source dataset that is mapped
|
|
to the virtual selection.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Virtual Selection \#n <em>(variable size)</em></td>
|
|
<td>This is the @ref FMT4DataspaceSEL "dataspace selection" in the virtual dataset that
|
|
is mapped to the source selection.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the block.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_infra_fractalheap III.G. Disk Format: Level 1G - Fractal Heap
|
|
Each fractal heap consists of a header and zero or more direct and indirect blocks (described below).
|
|
The header contains general information as well as initialization parameters for the doubling
|
|
table. The <em>Address of Root Block</em> field in the header points to the first direct or indirect block in
|
|
the heap.
|
|
|
|
Fractal heaps are based on a data structure called a <em>doubling table</em>. A doubling table provides
|
|
a mechanism for quickly extending an array-like data structure that minimizes the number of empty blocks
|
|
in the heap, while retaining very fast lookup of any element within the array. More information on
|
|
fractal heaps and doubling tables can be found in the RFC
|
|
“\ref_rfc20070115 .”
|
|
|
|
The fractal heap implements the doubling table structure with indirect and direct blocks. Indirect
|
|
blocks in the heap do not actually contain data for objects in the heap, their “size” is
|
|
abstract - they represent the indexing structure for locating the direct blocks in the doubling table.
|
|
Direct blocks contain the actual data for objects stored in the heap.
|
|
|
|
All indirect blocks have a constant number of block entries in each row, called the <em>width</em>
|
|
of the doubling table (see <em>Table Width</em> field in the header). The number of rows for each indirect
|
|
block in the heap is determined by the size of the block that the indirect block represents in the doubling table
|
|
(calculation of this is shown below) and is constant, except for the “root” indirect block,
|
|
which expands and shrinks its number of rows as needed.
|
|
|
|
Blocks in the first <em>two</em> rows of an indirect block are <em>Starting Block Size</em> number of
|
|
bytes in size. For example, if the row <em>width</em> of the doubling table is 4, then the first eight block
|
|
entries in the indirect block are <em>Starting Block Size</em> number of bytes in size. The blocks in each
|
|
subsequent row are twice the size of the blocks in the previous
|
|
row. In other words, blocks in the third row are twice the <em>Starting Block Size</em>, blocks in the
|
|
fourth row are four times the <em>Starting Block Size</em>, and so on. Entries for blocks up to the
|
|
<em>Maximum Direct Block Size</em> point to direct blocks, and entries for blocks greater than that size
|
|
point to further indirect blocks (which have their own entries for direct and indirect blocks). <em>Starting
|
|
Block Size</em> and <em>Maximum Direct Block Size</em> are fields stored in the header.
|
|
|
|
The number of rows of blocks, <em>nrows</em>, in an indirect block is calculated
|
|
by the following expression: <br /><br />
|
|
<em>nrows</em> = (log<sub>2</sub>(<em>iblock_size</em>) - log<sub>2</sub>(<em><Starting Block Size></em>)) + 1
|
|
where <em>block_size</em> is the size of the block that the indirect block
|
|
represents in the doubling table. For example, to represent a block with <em>block_size</em> equals to 1024,
|
|
and <em>Starting Block Size</em> equals to 256, three rows are needed.
|
|
|
|
The maximum number of rows of direct blocks, <em>max_dblock_rows</em>, in any indirect block of a fractal
|
|
heap is given by the following expression: <br /><br />
|
|
<em>max_dblock_rows</em> = (log<sub>2</sub>(<em><Maximum Direct Block Size></em>) -
|
|
log<sub>2</sub>(<em><Starting Block Size></em>)) + 2
|
|
|
|
Using the computed values for <em>nrows</em> and <em>max_dblock_rows</em>, along with the <em>Width</em>
|
|
of the doubling table, the number of direct and indirect block entries (<em>K</em> and <em>N</em> in the
|
|
indirect block description, below) in an indirect block can be computed: <br /> <br />
|
|
<em>K</em> = MIN(<em>nrows</em>, <em>max_dblock_rows</em>) * <em>Table Width</em> <br /><br />
|
|
If <em>nrows</em> is less than or equal to <em>max_dblock_rows</em>, <em>N</em> is 0. Otherwise, <em>N</em>
|
|
is simply computed: <br /><br />
|
|
<em>N</em> = <em>K</em> - (<em>max_dblock_rows</em> * <em>Table Width</em>)
|
|
|
|
The size of indirect blocks on disk is determined by the number of rows in the indirect block
|
|
(computed above). The size of direct blocks on disk is exactly the size of the block in the doubling table.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap Header</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Heap ID Length</td>
|
|
<td colspan="2">I/O Filters’ Encoded Length</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Maximum Size of Managed Objects</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Next Huge Object ID<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />v2 B-tree Address of Huge Objects<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Amount of Free Space in Managed Blocks<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of Managed Block Free Space Manager<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Amount of Managed Space in Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Amount of Allocated Managed Space in Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Offset of Direct Block Allocation Iterator in Managed Space<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Managed Objects in Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Huge Objects in Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Huge Objects in Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Tiny Objects in Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Tiny Objects in Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Table Width</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space insertedonly to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Starting Block Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Maximum Direct Block Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Maximum Heap Size</td>
|
|
<td colspan="2">Starting \# of Rows in Root Indirect Block</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of Root Block<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Current \# of Rows in Root Indirect Block</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Filtered Root Direct Block <em>(optional)</em><sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">I/O Filter Mask<em> (optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">I/O Filter Information<em> (optional, variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap Header</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>FRHP</code>” is used to indicate the beginning of a
|
|
fractal heap header. This gives file consistency checking utilities a better chance of reconstructing
|
|
a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Heap ID Length</td>
|
|
<td>This is the length in bytes of heap object IDs for this heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>I/O Filters’ Encoded Length</td>
|
|
<td>This is the size in bytes of the encoded <em>I/O Filter Information</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field is the heap status flag and is a bit field indicating additional information about
|
|
the fractal heap.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit(s)</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, the ID value to use for huge object has wrapped around. If the value for the
|
|
<em>Next Huge Object ID</em> has wrapped around, each new huge object inserted into the
|
|
heap will require a search for an ID value.
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, the direct blocks in the heap are checksummed.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-7</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Maximum Size of Managed Objects</td>
|
|
<td>This is the maximum size of managed objects allowed in the heap. Objects greater than this this
|
|
are ‘huge’ objects and will be stored in the file directly, rather than in a direct
|
|
block for the heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Next Huge Object ID</td>
|
|
<td>This is the next ID value to use for a huge object in the heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>v2 B-tree Address of Huge Objects</td>
|
|
<td>This is the address of the @ref subsubsec_fmt4_infra_btrees_v2 used to track huge objects in the heap.
|
|
The type of records stored in the <em>v2 B-tree</em> will be determined by whether the address and
|
|
length of a huge object can fit into a heap ID (if yes, it is a “directly” accessed huge
|
|
object) and whether there is a filter used on objects in the heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Amount of Free Space in Managed Blocks</td>
|
|
<td>This is the total amount of free space in managed direct blocks (in bytes).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address of Managed Block Free Space Manager</td>
|
|
<td>This is the address of the <em>@ref subsec_fmt4_infra_freespaceindex</em>
|
|
for managed blocks.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Amount of Managed Space in Heap</td>
|
|
<td>This is the total amount of managed space in the heap (in bytes), essentially the
|
|
upper bound of the heap’s linear address space.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Amount of Allocated Managed Space in Heap</td>
|
|
<td>This is the total amount of managed space (in bytes) actually allocated in the heap.
|
|
This can be less than the <em>Amount of Managed Space in Heap</em> field, if some direct
|
|
blocks in the heap’s linear address space are not allocated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Offset of Direct Block Allocation Iterator in Managed Space</td>
|
|
<td>This is the linear heap offset where the next direct block should be allocated at (in bytes).
|
|
This may be less than the <em>Amount of Managed Space in Heap</em> value because the heap’s
|
|
address space is increased by a “row” of direct blocks at a time, rather than by single
|
|
direct block increments.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Managed Objects in Heap</td>
|
|
<td>This is the number of managed objects in the heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Huge Objects in Heap</td>
|
|
<td>This is the total size of huge objects in the heap (in bytes).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Huge Objects in Heap</td>
|
|
<td>This is the number of huge objects in the heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Tiny Objects in Heap</td>
|
|
<td>This is the total size of tiny objects that are packed in heap IDs (in bytes).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Tiny Objects in Heap</td>
|
|
<td>This is the number of tiny objects that are packed in heap IDs.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Table Width</td>
|
|
<td>This is the number of columns in the doubling table for managed blocks. This value
|
|
must be a power of two.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Starting Block Size</td>
|
|
<td>This is the starting block size to use in the doubling table for managed blocks (in bytes).
|
|
This value must be a power of two.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Maximum Direct Block Size</td>
|
|
<td>This is the maximum size allowed for a managed direct block. Objects inserted into the heap that
|
|
are larger than this value (less the number of bytes of direct block prefix/suffix) are stored as
|
|
‘huge’ objects. This value must be a power of two.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Maximum Heap Size</td>
|
|
<td>This is the maximum size of the heap’s linear address space for managed objects (in bytes).
|
|
The value stored is the log2 of the actual value, that is: the number of bits of the address space.
|
|
‘Huge’ and ‘tiny’ objects are not counted in this value, since they do not
|
|
store objects in the linear address space of the heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Starting \# of Rows in Root Indirect Block</td>
|
|
<td>This is the starting number of rows for the root indirect block. A value of 0 indicates that the
|
|
root indirect block will have the maximum number of rows needed to address the heap’s
|
|
<em>Maximum Heap Size</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address of Root Block</td>
|
|
<td>This is the address of the root block for the heap. It can be the
|
|
@ref FMT4UndefinedAddress "undefined address" if there is no data in the heap. It either
|
|
points to a direct block (if the <em>Current \# of Rows in the Root Indirect Block</em> value is 0),
|
|
or an indirect block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Current \# of Rows in Root Indirect Block</td>
|
|
<td>This is the current number of rows in the root indirect block. A value of 0 indicates that
|
|
<em>Address of Root Block</em> points to direct block instead of indirect block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Filtered Root Direct Block</td>
|
|
<td>This is the size of the root direct block, if filters are applied to heap objects (in bytes).
|
|
This field is only stored in the header if the <em>I/O Filters’ Encoded Length</em> is
|
|
greater than 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>I/O Filter Mask</td>
|
|
<td>This is the filter mask for the root direct block, if filters are applied to heap objects. This
|
|
mask has the same format as that used for the filter mask in chunked raw data records in a
|
|
@ref subsubsec_fmt4_infra_btrees_v1. This field is only stored in the header if the <em>I/O Filters’
|
|
Encoded Length</em> is greater than 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>I/O Filter Information</td>
|
|
<td>This is the I/O filter information encoding direct blocks and huge objects, if filters are applied to
|
|
heap objects. This field is encoded as a @ref subsubsec_fmt4_dataobject_hdr_msg_filter message. The size
|
|
of this field is determined by <em>I/O Filters’ Encoded Length</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the header.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap Direct Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Heap Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Block Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Object Data <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap Direct Block</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>FHDB</code>” is used to indicate the beginning of
|
|
a fractal heap direct block. This gives file consistency checking utilities a better chance of
|
|
reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Heap Header Address</td>
|
|
<td>This is the address for the fractal heap header that this block belongs to. This field is
|
|
principally used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Block Offset</td>
|
|
<td>This is the offset of the block within the fractal heap’s address space (in bytes). The
|
|
number of bytes used to encode this field is the <em>Maximum Heap Size</em> (in the heap’s
|
|
header) divided by 8 and rounded up to the next highest integer, for values that are not a multiple
|
|
of 8. This value is principally used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the direct block. This field is only present if bit 1 of <em>Flags</em>
|
|
in the heap’s header is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Object Data</td>
|
|
<td>This section of the direct block stores the actual data for objects in the heap. The size of this
|
|
section is determined by the direct block’s size minus the size of the other fields stored in the
|
|
direct block (for example, the <em>Signature</em>, <em>Version</em>, and others including the
|
|
<em>Checksum</em> if it is present).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap Indirect Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Heap Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Block Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Direct Block \#0 Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Filtered Direct Block \#0 <em>(optional)</em> <sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask for Direct Block \#0 <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Direct Block \#1 Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Filtered Direct Block \#1 <em>(optional)</em><sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask for Direct Block \#1 <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Direct Block \#K-1 Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Filtered Direct Block \#K-1 <em>(optional)</em><sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask for Direct Block \#K-1 <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Indirect Block \#0 Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Indirect Block \#1 Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Child Indirect Block \#N-1 Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap Indirect Block</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>FHIB</code>” is used to indicate the beginning of a
|
|
fractal heap indirect block. This gives file consistency checking utilities a better chance of
|
|
reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Heap Header Address</td>
|
|
<td>This is the address for the fractal heap header that this block belongs to. This field is principally
|
|
used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Block Offset</td>
|
|
<td>This is the offset of the block within the fractal heap’s address space (in bytes). The number
|
|
of bytes used to encode this field is the <em>Maximum Heap Size</em> (in the heap’s header)
|
|
divided by 8 and rounded up to the next highest integer, for values that are not a multiple of 8. This
|
|
value is principally used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Child Direct Block \#K Address</td>
|
|
<td>This field is the address of the child direct block. The size of the [uncompressed] direct block can
|
|
be computed by its offset in the heap’s linear address space.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Filtered Direct Block \#K</td>
|
|
<td>This is the size of the child direct block after passing through the I/O filters defined for this heap
|
|
(in bytes). If no I/O filters are present for this heap, this field is not present.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filter Mask for Direct Block \#K</td>
|
|
<td>This is the I/O filter mask for the filtered direct block. This mask has the same format as that
|
|
used for the filter mask in chunked raw data records in a @ref subsubsec_fmt4_infra_btrees_v1. If
|
|
no I/O filters are present for this heap, this field is not present.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Child Indirect Block \#N Address</td>
|
|
<td>This field is the address of the child indirect block. The size of the indirect block can be computed
|
|
by its offset in the heap’s linear address space.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the indirect block.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
An object in the fractal heap is identified by means of a fractal heap ID, which encodes information to
|
|
locate the object in the heap. Currently, the fractal heap stores an object in one of three ways,
|
|
depending on the object’s size:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Type</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">Tiny</td>
|
|
<td>When an object is small enough to be encoded in the heap ID, the object’s data is embedded
|
|
in the fractal heap ID itself. There are two sub-types for this type of object: normal and extended.
|
|
The sub-type for tiny heap IDs depends on whether the heap ID is large enough to store objects
|
|
greater than 16 bytes or not. If the heap ID length is 18 bytes or smaller, the ‘normal’
|
|
tiny heap ID form is used. If the heap ID length is greater than 18 bytes in length, the
|
|
“extended” form is used. See format description below for both sub-types.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">Huge</td>
|
|
<td>When the size of an object is larger than <em>Maximum Size of Managed Objects</em> in the
|
|
<em>Fractal Heap Header</em>, the object’s data is stored on its own in the file and the object
|
|
is tracked/indexed via a version 2 B-tree. All huge objects for a particular fractal heap use the same
|
|
v2 B-tree. All huge objects for a particular fractal heap use the same format for their huge object IDs.
|
|
<br />Depending on whether the IDs for a heap are large enough to hold the object’s retrieval
|
|
information and whether I/O pipeline filters are applied to the heap’s objects, 4 sub-types are
|
|
derived for huge object IDs for this heap:
|
|
<table>
|
|
<tr>
|
|
<th align="left" width="35%">Sub-type</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Directly accessed, non-filtered</td>
|
|
<td>The object’s address and length are embedded in the fractal heap ID itself and the
|
|
object is directly accessed from them. This allows the object to be accessed without resorting
|
|
to the B-tree.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Directly accessed, filtered</td>
|
|
<td>The filtered object’s address, length, filter mask and de-filtered size are embedded
|
|
in the fractal heap ID itself and the object is accessed directly with them. This allows the
|
|
object to be accessed without resorting to the B-tree.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Indirectly accessed, non-filtered</td>
|
|
<td>The object is located by using a B-tree key embedded in the fractal heap ID to retrieve the
|
|
address and length from the version 2 B-tree for huge objects. Then, the address and length
|
|
are used to access the object.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Indirectly accessed, filtered</td>
|
|
<td>The object is located by using a B-tree key embedded in the fractal heap ID to retrieve the
|
|
filtered object’s address, length, filter mask and de-filtered size from the version
|
|
2 B-tree for huge objects. Then, this information is used to access the object.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center">Managed</td>
|
|
<td>When the size of an object does not meet the above two conditions, the object is stored and managed
|
|
via the direct and indirect blocks based on the doubling table.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The specific format for each type of heap ID is described below:
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap ID for Tiny Objects (sub-type 1 - ‘Normal’)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version, Type & Length</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data <em>(variable size)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap ID for Tiny Objects (sub-type 1 - ‘Normal’)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version, Type, and Length</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>The current version of ID format. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4-5</code></td>
|
|
<td>The ID type. Tiny objects have a value of <code>2</code>.
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-3</code></td>
|
|
<td>The length of the tiny object. The value stored is one less than the actual length (since
|
|
zero-length objects are not allowed to be stored in the heap). For example, an object of
|
|
actual length 1 has an encoded length of 0, an object of actual length 2 has an encoded
|
|
length of 1, and so on.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data</td>
|
|
<td>This is the data for the object.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap ID for Tiny Objects (sub-type 2 - ‘Extended’)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version, Type, and Length</td>
|
|
<td>Extended Length</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Data <em>(variable size)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap ID for Tiny Objects (sub-type 2 - ‘Extended’)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version, Type, and Length</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>The current version of ID format. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4-5</code></td>
|
|
<td>The ID type. Tiny objects have a value of <code>2</code>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-3</code></td>
|
|
<td>These 4 bits, together with the next byte, form an unsigned 12-bit integer for holding the
|
|
length of the object. These 4-bits are bits 8-11 of the 12-bit integer. See description
|
|
for the <em>Extended Length</em> field below.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Extended Length</td>
|
|
<td>This byte, together with the 4 bits in the previous byte, forms an unsigned 12-bit integer for
|
|
holding the length of the tiny object. These 8 bits are bits 0-7 of the 12-bit integer formed. The
|
|
value stored is one less than the actual length (since zero-length objects are not allowed to be
|
|
stored in the heap). For example, an object of actual length 1 has an encoded length of 0, an object of
|
|
actual length 2 has an encoded length of 1, and so on.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data</td>
|
|
<td>This is the data for the object.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap ID for Huge Objects (sub-type 1 & 2): indirectly accessed,
|
|
non-filtered/filtered</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />v2 B-tree Key<sup>L</sup><em> (variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap ID for Huge Objects (sub-type 1 & 2): indirectly accessed,
|
|
non-filtered/filtered</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>The current version of ID format. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4-5</code></td>
|
|
<td>The ID type. Huge objects have a value of <code>1</code>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-3</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>v2 B-tree Key</td>
|
|
<td>This field is the B-tree key for retrieving the information from the version 2 B-tree for huge
|
|
objects needed to access the object. See the description of @ref subsubsec_fmt4_infra_btrees_v2
|
|
records sub-type 1 & 2 for a description of the fields. New key values are derived from <em>Next
|
|
Huge Object ID</em> in the <em>Fractal Heap Header</em>.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap ID for Huge Objects (sub-type 3): directly accessed, non-filtered</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address <sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Length <sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap ID for Huge Objects (sub-type 3): directly accessed, non-filtered</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>The current version of ID format. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4-5</code></td>
|
|
<td>The ID type. Huge objects have a value of <code>1</code>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-3</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This field is the address of the object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length</td>
|
|
<td>This field is the length of the object in the file.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap ID for Huge Objects (sub-type 4): directly accessed, filtered</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address <sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Length <sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />De-filtered Size <sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap ID for Huge Objects (sub-type 4): directly accessed, filtered</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>The current version of ID format. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4-5</code></td>
|
|
<td>The ID type. Huge objects have a value of <code>1</code>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-3</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This field is the address of the filtered object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length</td>
|
|
<td>This field is the length of the filtered object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filter Mask</td>
|
|
<td>This field is the I/O pipeline filter mask for the filtered object in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filtered Size</td>
|
|
<td>This field is the size of the de-filtered object in the file.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap ID for Managed Objects</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Length <em>(variable size)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap ID for Managed Objects</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version and Type</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>The current version of ID format. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4-5</code></td>
|
|
<td>The ID type. Managed objects have a value of <code>0</code>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-3</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Offset</td>
|
|
<td>This field is the offset of the object in the heap. This field’s size is the minimum number of
|
|
bytes necessary to encode the <em>Maximum Heap Size</em> value (from the <em>Fractal Heap Header</em>).
|
|
For example, if the value of the <em>Maximum Heap Size</em> is less than 256 bytes, this field is 1
|
|
byte in length, a <em>Maximum Heap Size</em> of 256-65535 bytes uses a 2 byte length, and so on.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length</td>
|
|
<td>This field is the length of the object in the heap. It is determined by taking the minimum value
|
|
of <em>Maximum Direct Block Size</em> and <em>Maximum Size of Managed Objects</em> in the <em>Fractal
|
|
Heap Header</em>. Again, the minimum number of bytes needed to encode that value is used for the size
|
|
of this field.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_infra_freespaceindex III.H. Disk Format: Level 1H - Free-space Index
|
|
Free-space managers are used to describe space within a heap or the entire HDF5 file that is not currently
|
|
used for that heap or file.
|
|
|
|
The <em>free-space manager header</em> contains metadata information about the space being tracked, along
|
|
with the address of the list of <em>free space sections</em> which actually describes the free space. The
|
|
header records information about free-space sections being tracked, creation parameters for handling
|
|
free-space sections of a client, and section information used to locate the collection of free-space sections.
|
|
|
|
The <em>free-space section list</em> stores a collection of free-space sections that is specific to each
|
|
<em>client</em> of the free-space manager. For example, the fractal heap is a client of the free space
|
|
manager and uses it to track unused space within the heap. There are 4 types of section records for the
|
|
fractal heap, each of which has its own format, listed below.
|
|
<table>
|
|
<caption><strong>Layout: Free-space Manager Header</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Client ID</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Total Space Tracked<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Total Number of Sections<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Serialized Sections<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Number of Un-Serialized Sections<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Number of Section Classes</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Shrink Percent</td>
|
|
<td colspan="2">Expand Percent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Size of Address Space</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Maximum Section Size <sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of Serialized Section List<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Serialized Section List Used<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Allocated Size of Serialized Section List<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Free-space Manager Header</strong></caption>
|
|
<tr>
|
|
<th width="35%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>FSHD</code>” is used to indicate the beginning of the
|
|
Free-space Manager Header. This gives file consistency checking utilities a better chance of
|
|
reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This is the version number for the Free-space Manager Header and this document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Client ID</td>
|
|
<td>This is the client ID for identifying the user of this free-space manager:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Fractal heap</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>File</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Total Space Tracked</td>
|
|
<td>This is the total amount of free space being tracked, in bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Total Number of Sections</td>
|
|
<td>This is the total number of free-space sections being tracked.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Serialized Sections</td>
|
|
<td>This is the number of serialized free-space sections being tracked.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Un-Serialized Sections</td>
|
|
<td>This is the number of un-serialized free-space sections being managed. Un-serialized sections are
|
|
created by the free-space client when the list of sections is read in.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Section Classes</td>
|
|
<td>This is the number of section classes handled by this free space manager for the free-space client.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Shrink Percent</td>
|
|
<td>This is the percent of current size to shrink the allocated serialized free-space section list.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Expand Percent</td>
|
|
<td>This is the percent of current size to expand the allocated serialized free-space section list.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Address Space</td>
|
|
<td>This is the size of the address space that free-space sections are within. This is stored as the
|
|
log<sub>2</sub> of the actual value (in other words, the number of bits required to store values
|
|
within that address space).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Maximum Section Size</td>
|
|
<td>This is the maximum size of a section to be tracked.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address of Serialized Section List</td>
|
|
<td>This is the address where the serialized free-space section list is stored.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Serialized Section List Used</td>
|
|
<td>This is the size of the serialized free-space section list used (in bytes). This value must be
|
|
less than or equal to the <em>allocated size of serialized section list</em>, below.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Allocated Size of Serialized Section List</td>
|
|
<td>This is the size of serialized free-space section list actually allocated (in bytes).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the free-space manager header.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The free-space sections being managed are stored in a <em>free-space section list</em>, described below.
|
|
The sections in the free-space section list are stored in the following way: a count of the number of sections
|
|
describing a particular size of free space and the size of the free-space described (in bytes), followed
|
|
by a list of section description records; then another section count and size, followed by the list of
|
|
section descriptions for that size; and so on.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Free-space Section List</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Free-space Manager Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Number of Section Records in Set \#0 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Size of Free-space Section Described in Record Set \#0 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#0 Section Record \#0 Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Record Set \#0 Section Record #0 Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#0 Section Record \#0 Data <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#0 Section Record \#K-1 Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Record Set \#0 Section Record \#K-1 Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#0 Section Record \#K-1 Data <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Number of Section Records in Set \#1 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Size of Free-space Section Described in Record Set \#1 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#1 Section Record \#0 Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Record Set \#1 Section Record \#0 Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#1 Section Record \#0 Data <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#1 Section Record \#K-1 Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Record Set \#1 Section Record \#K-1 Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#1 Section Record \#K-1 Data <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><strong>...</strong></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><strong>...</strong></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Number of Section Records in Set \#N-1 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Size of Free-space Section Described in Record Set \#N-1 <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#N-1 Section Record \#0 Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Record Set \#N-1 Section Record \#0 Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#N-1 Section Record \#0 Data <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#N-1 Section Record \#K-1 Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Record Set \#N-1 Section Record \#K-1 Type</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Record Set \#N-1 Section Record \#K-1 Data <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Free-space Section List</strong></caption>
|
|
<tr>
|
|
<th width="35%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>FSSE</code>” is used to indicate the beginning of the
|
|
Free-space Section Information. This gives file consistency checking utilities a better chance of
|
|
reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This is the version number for the Free-space Section List and this document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Free-space Manager Header Address</td>
|
|
<td>This is the address of the <em>Free-space Manager Header</em>. This field is principally used for file
|
|
integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Section Records for Set \#N</td>
|
|
<td>This is the number of free-space section records for set \#N. The length of this field is the minimum
|
|
number of bytes needed to store the <em>number of serialized sections</em> (from the <em>free-space
|
|
manager header</em>).<br />
|
|
The number of sets of free-space section records is determined by the <em>size of serialized section
|
|
list</em> in the <em>free-space manager header</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Section Size for Record Set \#N</td>
|
|
<td>This is the size (in bytes) of the free-space section described for <em>all</em> the section records
|
|
in set \#N.<br />
|
|
The length of this field is the minimum number of bytes needed to store the <em>maximum section
|
|
size</em> (from the <em>free-space manager header</em>).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Record Set \#N Section \#K Offset</td>
|
|
<td>This is the offset (in bytes) of the free-space section within the client for the free-space manager.
|
|
<br />The length of this field is the minimum number of bytes needed to store the <em>size of address
|
|
space</em> (from the <em>free-space manager header</em>).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Record Set \#N Section \#K Type</td>
|
|
<td>This is the type of the section record, used to decode the <em>record set \#N section \#K data</em>
|
|
information. The defined record type for <em>file</em> client is:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Type</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>File’s section (a range of actual bytes in file)</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table>
|
|
<br />The defined record types for a <em>fractal heap</em> client are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Type</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Fractal heap “single” section</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Fractal heap “first row” section</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Fractal heap “normal row” section</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Fractal heap “indirect” section</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Record Set \#N Section \#K Data</td>
|
|
<td>This is the section-type specific information for each record in the record set, described below.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the <em>Free-space Section List</em>.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The section-type specific data for each free-space section record is described below:
|
|
<table>
|
|
<caption><strong>Layout: File’s Section Data Record</strong></caption>
|
|
<tr>
|
|
<td colspan="4"><em>No additional record data stored</em></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap “Single” Section Data Record</strong></caption>
|
|
<tr>
|
|
<td colspan="4"><em>No additional record data stored</em></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap “First Row” Section Data Record</strong></caption>
|
|
<tr>
|
|
<td colspan="4"><em>Same format as “indirect” section data</em></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap “Normal Row” Section Data Record</strong></caption>
|
|
<tr>
|
|
<td colspan="4"><em>No additional record data stored</em></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Layout: Fractal Heap “Indirect” Section Data Record</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Fractal Heap Indirect Block Offset <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Block Start Row</td>
|
|
<td colspan="2">Block Start Column</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Number of Blocks</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fractal Heap “Indirect” Section Data Record</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Fractal Heap Block Offset</td>
|
|
<td>The offset of the indirect block in the fractal heap’s address space containing the empty
|
|
blocks.<br />
|
|
The number of bytes used to encode this field is the minimum number of bytes needed to encode
|
|
values for the <em>Maximum Heap Size</em> (in the fractal heap’s header).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Block Start Row</td>
|
|
<td>This is the row that the empty blocks start in.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Block Start Column</td>
|
|
<td>This is the column that the empty blocks start in.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Blocks</td>
|
|
<td>This is the number of empty blocks covered by the section.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_infra_sohm III.I. Disk Format: Level 1I - Shared Object Header Message (SOHM) Master Table
|
|
The <em>shared object header message table</em> is used to locate object header messages that are shared
|
|
between two or more object headers in the file. Shared object header messages are stored and indexed in
|
|
the file in one of two ways: indexed sequentially in a <em>shared header message list</em> or indexed
|
|
with a v2 B-tree. The shared messages themselves are either stored in a fractal heap (when two or more
|
|
objects share the message), or remain in an object’s header (when only one object uses the message
|
|
currently, but the message can be shared in the future).
|
|
|
|
The <em>shared object header message table</em> contains a list of shared message index headers. Each
|
|
index header records information about the version of the index format, the index storage type, flags
|
|
for the message types indexed, the number of messages in the index, the address where the index resides,
|
|
and the fractal heap address if shared messages are stored there.
|
|
|
|
Each index can be either a list or a v2 B-tree and may transition between those two forms as the number
|
|
of messages in the index varies. Each shared message record contains information used to locate the
|
|
shared message from either a fractal heap or an object header. The types of messages that can be shared
|
|
are: <em>Dataspace, Datatype, Fill Value, Filter Pipeline and Attribute</em>.
|
|
|
|
The <em>shared object header message master table</em> is pointed to from a
|
|
@ref subsubsec_fmt4_dataobject_hdr_msg_shared in the superblock extension for a file. This
|
|
message stores the version of the table format, along with the number of index headers, \a N, in
|
|
the table.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Shared Object Header Message (SOHM) Master Table</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version for index \#0</td>
|
|
<td>Index Type for index #0</td>
|
|
<td colspan="2">Message Type Flags for index \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Minimum Message Size for index \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">List Cutoff for index \#0</td>
|
|
<td colspan="2">v2 B-tree Cutoff for index \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Number of Messages for index \#0</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Index Address<sup>O</sup> for index \#0<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Fractal Heap Address<sup>O</sup> for index \#0<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version for index \#N-1</td>
|
|
<td>Index Type for index \#N-1</td>
|
|
<td colspan="2">Message Type Flags for index \#N-1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Minimum Message Size for index \#N-1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">List Cutoff for index \#N-1</td>
|
|
<td colspan="2">v2 B-tree Cutoff for index \#N-1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Number of Messages for index \#N-1</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Index Address<sup>O</sup> for index \#N-1<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Fractal Heap Address<sup>O</sup> for index \#N-1<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Shared Object Header Message (SOHM) Master Table</strong></caption>
|
|
<tr>
|
|
<th width="35%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>SMTB</code>” is used to indicate the beginning of the
|
|
Shared Object Header Message table. This gives file consistency checking utilities a better chance
|
|
of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version for index \#N</td>
|
|
<td>This is the version number for the list of shared object header message indexes and this document
|
|
describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Index Type for index \#N</td>
|
|
<td>The type of index can be an unsorted list or a v2 B-tree.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Type Flags for index \#N</td>
|
|
<td>This field indicates the type of messages tracked in the index, as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bits</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, the index tracks <em>Dataspace Messages</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, the message tracks <em>Datatype Messages</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>If set, the message tracks <em>Fill Value Messages</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>If set, the message tracks <em>Filter Pipeline Messages</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>If set, the message tracks <em>Attribute Messages</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5-15</code></td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
An index can track more than one type of message, but each type of message can only by in one index.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Minimum Message Size for index \#N</td>
|
|
<td>This is the message size sharing threshold for the index. If the encoded size of the message is
|
|
less than this value, the message is not shared.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>List Cutoff for index \#N</td>
|
|
<td>This is the cutoff value for the indexing of messages to switch from a list to a v2 B-tree. If the
|
|
number of messages is greater than this value, the index should be a v2 B-tree.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>v2 B-tree Cutoff for index \#N</td>
|
|
<td>This is the cutoff value for the indexing of messages to switch from a v2 B-tree back to a list.
|
|
If the number of messages is less than this value, the index should be a list.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Messages for index \#N</td>
|
|
<td>The number of shared messages being tracked for the index.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Index Address for index \#N</td>
|
|
<td>This field is the address of the list or v2 B-tree where the index nodes reside.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Fractal Heap Address for index \#N</td>
|
|
<td>This field is the address of the fractal heap if shared messages are stored there.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the table.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
Shared messages are indexed either with a <em>shared message record list</em>, described below,
|
|
or using a v2 B-tree (using record type 7). The number of records, \a M, in the <em>shared message
|
|
record list</em> is determined in the index’s entry in the <em>shared object header message
|
|
table</em> (see “Number of Messages for index \#N” above).
|
|
<table>
|
|
<caption><strong>Layout: Shared Message Record List</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Shared Message Record \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Shared Message Record \#1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Shared Message Record \#M-1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Shared Message Record List</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>SMLI</code>” is used to indicate the beginning of a
|
|
list of index nodes. This gives file consistency checking utilities a better chance of
|
|
reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Shared Message Record \#M</td>
|
|
<td>The record for locating the shared message, either in the fractal heap for the index, or an object
|
|
header (see format for <em>index nodes</em> below).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the list.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The record for each shared message in an index is stored in one of the following forms:
|
|
<table>
|
|
<caption><strong>Layout: Shared Message Record, for Messages Stored in a Fractal Heap</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Location</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Hash Value</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Reference Count</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Fractal Heap ID<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Shared Message Record, for Messages Stored in a Fractal Heap</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Location</td>
|
|
<td>This has a value of 0 indicating that the message is stored in the heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Hash Value</td>
|
|
<td>This is the hash value for the message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Reference Count</td>
|
|
<td>This is the number of times the message is used in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Fractal Heap ID</td>
|
|
<td>This is an 8-byte fractal heap ID for the message as stored in the fractal heap for the index.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Shared Message Record, for Messages Stored in an Object Header</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Location</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Hash Value</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Reserved</td>
|
|
<td>Message Type</td>
|
|
<td colspan="2">Creation Index</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Object Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Shared Message Record, for Messages Stored in an Object Header</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Location</td>
|
|
<td>This has a value of 1 indicating that the message is stored in an object header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Hash Value</td>
|
|
<td>This is the hash value for the message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Message Type</td>
|
|
<td>This is the message type in the object header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Creation Index</td>
|
|
<td>This is the creation index of the message within the object header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Object Header Address</td>
|
|
<td>This is the address of the object header where the message is located.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_infra_mdci III.J. Disk Format: Level 1J - Metadata Cache Image Block
|
|
The <em>metadata cache image block</em> is a serialized copy of the contents of the metadata cache at file
|
|
close, written when generation of a metadata cache image is enabled for the file (see
|
|
#H5Pset_mdc_image_config). It allows the library to reconstruct the cache directly from this block the
|
|
next time the file is opened, instead of reconstructing it by parsing the file’s metadata, which can
|
|
significantly reduce the time needed to open a file containing many pieces of cached metadata. The block
|
|
is pointed to by the @ref subsubsec_fmt4_dataobject_hdr_msg_mdci in the superblock extension.
|
|
|
|
The block begins with a signature, version, flags, image data length, and entry count, followed by one
|
|
record for each metadata cache entry captured in the image, and ends with a checksum for the block.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Metadata Cache Image Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Image Data Length<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Number of Entries</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Image Entry \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Image Entry \#1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Image Entry \#N-1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Metadata Cache Image Block</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>MDCI</code>” is used to indicate the beginning of the
|
|
Metadata Cache Image Block. This gives file consistency checking utilities a better chance of
|
|
reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This is the version number for the Metadata Cache Image Block and this document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field holds flags for the block, as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bits</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, adaptive cache resize status information follows the entries, below. The library does
|
|
not currently generate this information, so this bit is always <code>0</code>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1-7</code></td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Image Data Length</td>
|
|
<td>This is the total length, in bytes, of the metadata cache image block, including this header, all
|
|
entries, and the trailing checksum. This is the same value stored in the “Metadata Cache Image
|
|
Block Size” field of the @ref subsubsec_fmt4_dataobject_hdr_msg_mdci that points to this
|
|
block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Entries</td>
|
|
<td>This is the number of metadata cache entry records, \a N, that follow the header. This value must be
|
|
greater than 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Image Entry \#i (i = 0 to (N-1))</td>
|
|
<td>These are the \a N metadata cache entry records captured in the image, one per entry that was
|
|
resident in the metadata cache at file close. The format of each entry is described below in
|
|
“Layout: Image Entry”.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for the entire metadata cache image block, computed over the header and all
|
|
\a N entries that precede it (that is, over the first “Image Data Length” minus 4 bytes
|
|
of the block).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Image Entry</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Type ID</td>
|
|
<td>Flags</td>
|
|
<td>Ring</td>
|
|
<td>Age</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Flush Dependency Child Count</td>
|
|
<td colspan="2">Flush Dependency Dirty Child Count</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Flush Dependency Parent Count</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">LRU Rank</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Entry Offset<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Entry Length<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Flush Dependency Parent Addresses<sup>O</sup> <em>(0 or more, per “Flush Dependency
|
|
Parent Count”, above)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Entry Image <em>(variable size, per “Entry Length”, above)</em></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Image Entry</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Type ID</td>
|
|
<td>This is an internal, implementation-defined identifier for the type of metadata cache client (for
|
|
example, superblock, object header, B-tree node, or fractal heap header) that owns this entry. The
|
|
library uses this value to determine how to reconstruct the entry in memory when the image is
|
|
loaded.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field holds flags for the entry, as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bits</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, the entry was dirty (not yet written to the file) at the time the image was
|
|
generated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, the entry was on the cache’s LRU list, and “LRU Rank”, below, gives
|
|
its position on that list.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>If set, the entry is the parent in one or more flush dependency relationships.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>If set, the entry is a child in one or more flush dependency relationships, and the entry
|
|
record is followed by one or more “Flush Dependency Parent Address” fields,
|
|
below.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4-7</code></td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Ring</td>
|
|
<td>This identifies the flush dependency “ring” the entry belongs to, which constrains the
|
|
order in which entries are flushed relative to entries in other rings (entries in an outer ring are
|
|
always flushed before entries in an inner ring), as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>User data (datasets and groups). Outermost ring; flushed first.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Raw data free-space manager.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Metadata free-space manager.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>Superblock extension.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>Superblock. Innermost ring; flushed last.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Age</td>
|
|
<td>This is the number of file opens and closes since the entry was last accessed by the application. The
|
|
library uses this value to decide whether to continue including the entry in future cache
|
|
images.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flush Dependency Child Count</td>
|
|
<td>This is the number of flush dependency children for which this entry is the parent. This is non-zero
|
|
only if bit 2 of “Flags”, above, is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flush Dependency Dirty Child Count</td>
|
|
<td>This is the number of flush dependency children, of the total given in “Flush Dependency Child
|
|
Count”, above, that were dirty at the time the image was generated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flush Dependency Parent Count</td>
|
|
<td>This is the number of flush dependency parents for which this entry is a child, \a K. This is
|
|
non-zero only if bit 3 of “Flags”, above, is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>LRU Rank</td>
|
|
<td>If bit 1 of “Flags”, above, is set, this is the entry’s position on the cache’s
|
|
LRU list (lower values are closer to the head of the list, i.e. more recently used). Otherwise, this
|
|
field has the value -1.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Entry Offset</td>
|
|
<td>This is the address in the file at which the entry’s metadata is (or will be) stored.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Entry Length</td>
|
|
<td>This is the length, in bytes, of the “Entry Image”, below.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flush Dependency Parent Address \#K</td>
|
|
<td>This is the file address of a flush dependency parent entry of this entry. There are as many of
|
|
these fields as given by “Flush Dependency Parent Count”, above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Entry Image</td>
|
|
<td>This is a byte-for-byte copy of the entry’s serialized on-disk image, as it would otherwise
|
|
appear at “Entry Offset”, above, in the file (for example, the disk format of an object
|
|
header, B-tree node, or other cached metadata structure).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\section sec_fmt4_dataobject IV. Disk Format: Level 2 - Data Objects
|
|
Data objects contain the “real” user-visible information in the file. These objects compose
|
|
the scientific data and other information which are generally thought of as “data” by the
|
|
end-user. All the other information in the file is provided as a framework for storing and accessing
|
|
these data objects.
|
|
|
|
A data object is composed of header and data information. The header information contains the
|
|
information needed to interpret the data information for the object as well as additional “metadata”
|
|
or pointers to additional “metadata” used to describe or annotate each object.
|
|
|
|
\subsection subsec_fmt4_dataobject_hdr IV.A. Disk Format: Level 2A - Data Object Headers
|
|
The header information of an object is designed to encompass all the information about an object, except for
|
|
the data itself. This information includes the dataspace, datatype, information about how the data is stored
|
|
on disk (in external files, compressed, broken up in blocks, and so on), as well as other information used by the
|
|
library to speed up access to the data objects or maintain a file’s integrity. Information stored by user
|
|
applications as attributes is also stored in the object’s header. The header of each object is not necessarily
|
|
located immediately prior to the object’s data in the file and in fact may be located in any position in the
|
|
file. The order of the messages in an object header is not significant.
|
|
|
|
Object headers are composed of a prefix and a set of messages. The prefix contains the information needed to
|
|
interpret the messages and a small amount of metadata about the object, and the messages contain the majority
|
|
of the metadata about the object.
|
|
|
|
\subsection subsec_fmt4_dataobject_hdr_prefix IV.A.1 Disk Format: Level 2A1 - Data Object Header Prefix
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_prefix_one IV.A.1.a Version 1 Data Object Header Prefix
|
|
Header messages are aligned on 8-byte boundaries for version 1 object headers.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Version 1 Object Header</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td width="25%">Version</td>
|
|
<td width="25%">Reserved (zero)</td>
|
|
<td colspan="2">Total Number of Header Messages</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Object Reference Count</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Object Header Size</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Header Message Type \#1</td>
|
|
<td colspan="2">Size of Header Message Data \#1</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Header Message \#1 Flags</td>
|
|
<td colspan="3">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Header Message Data \#1<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Header Message Type \#n</td>
|
|
<td colspan="2">Size of Header Message Data \#n</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Header Message \#n Flags</td>
|
|
<td colspan="3">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Header Message Data \#n<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Version 1 Object Header</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>This value is used to determine the format of the information in the object header. When the format of
|
|
the object header is changed, the version number is incremented and can be used to determine how the
|
|
information in the object header is formatted. This is version one (1) (there was no version zero (0))
|
|
of the object header.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Total Number of Header Messages</td>
|
|
<td>This value determines the total number of messages listed in object headers for this object. This value
|
|
includes the messages in continuation messages for this object.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Object Reference Count</td>
|
|
<td>This value specifies the number of “hard links” to this object within the current file.
|
|
References to the object from external files, “soft links” in this file and object
|
|
references in this file are not tracked.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Object Header Size</td>
|
|
<td>This value specifies the number of bytes of header message data following this length field that
|
|
contain object header messages for this object header. This value does not include the size of object header
|
|
continuation blocks for this object elsewhere in the file.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Header Message \#n Type</td>
|
|
<td>This value specifies the type of information included in the following header message data. The
|
|
message types for header messages are defined in sections below.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Size of Header Message \#n Data</td>
|
|
<td>This value specifies the number of bytes of header message data following the header message type,
|
|
length, flags, and reserved fields for the current message. The size includes padding bytes to make
|
|
the message a multiple of eight bytes.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Header Message \#n Flags</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Bit</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, the message data is constant. This is used for messages like the datatype message of
|
|
a dataset.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, the message is <em>shared</em> and stored in another location than the object header.
|
|
The Header Message Data field contains a Shared Message (described in the @ref
|
|
subsec_fmt4_dataobject_hdr_msg section below) and the Size of Header Message Data field contains
|
|
the size of that Shared Message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>If set, the message should not be shared.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>If set, the HDF5 decoder should fail to open this object if it does not understand the
|
|
message’s type and the file is open with permissions allowing write access to the file.
|
|
(Normally, unknown messages can just be ignored by HDF5 decoders)</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>If set, the HDF5 decoder should set bit 5 of this message’s flags (in other words, this
|
|
bit field) if it does not understand the message’s type and the object is modified in any
|
|
way. (Normally, unknown messages can just be ignored by HDF5 decoders)</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>If set, this object was modified by software that did not understand this message. (Normally,
|
|
unknown messages should just be ignored by HDF5 decoders) (Can be used to invalidate an index
|
|
or a similar feature)</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6</code></td>
|
|
<td>If set, this message is shareable.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>7</code></td>
|
|
<td>If set, the HDF5 decoder should always fail to open this object if it does not understand the
|
|
message’s type (whether it is open for read-only or read-write access). (Normally, unknown
|
|
messages can just be ignored by HDF5 decoders)</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Header Message \#n Data</td>
|
|
<td>The format and length of this field is determined by the header message type and size respectively.
|
|
Some header message types do not require any data and this information can be eliminated by setting the
|
|
length of the message to zero. The data is padded with enough zeros to make the size a multiple of
|
|
eight.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_prefix_two IV.A.1.b Version 2 Data Object Header Prefix
|
|
\anchor FMT4OhdrChunkDef A version 2 object header’s messages are stored in one or more <em>chunks</em>.
|
|
<em>Chunk \#0</em> is the chunk embedded directly in this prefix, below: it runs from the <code>OHDR</code>
|
|
signature through this chunk’s own <em>Gap</em> and <em>Checksum</em> fields. Additional chunks
|
|
(<em>chunk \#1</em> through <em>chunk \#n</em>) are stored in continuation blocks, each located and sized
|
|
by an Object Header Continuation message in some earlier chunk (see
|
|
@ref subsubsec_fmt4_dataobject_hdr_msg_continuation). Each chunk, including chunk \#0, ends with its
|
|
<em>own</em> <em>Gap</em> and <em>Checksum</em> fields, computed only over that chunk’s bytes. This
|
|
use of “chunk” is unrelated to the raw data <em>chunks</em> of a chunked-storage dataset (see
|
|
@ref subsubsec_fmt4_dataobject_hdr_msg_layout and @ref sec_fmt4_appendixc).
|
|
|
|
Note that the “total number of messages” field has been dropped from the data object header
|
|
prefix in this version. The number of messages in the data object header is just determined by the
|
|
messages encountered in all the object header blocks.
|
|
|
|
Note also that the fields and messages in this version of data object headers have <em>no</em> alignment
|
|
or padding bytes inserted - they are stored packed together.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Version 2 Object Header</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Access time <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Modification Time <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Change Time <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Birth Time <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Maximum \# of compact attributes <em>(optional)</em></td>
|
|
<td colspan="2">Minimum \# of dense attributes <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Chunk \#0 <em>(variable size)</em></td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message Type \#1</td>
|
|
<td colspan="2">Size of Header Message Data \#1</td>
|
|
<td>Header Message \#1 Flags</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Header Message \#1 Creation Order <em>(optional)</em></td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Message Data \#1<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message Type \#n</td>
|
|
<td colspan="2">Size of Header Message Data \#n</td>
|
|
<td>Header Message \#n Flags</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Header Message \#n Creation Order <em>(optional)</em></td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Message Data \#n<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Gap <em>(optional, variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Version 2 Object Header</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>OHDR</code>” is used to indicate the beginning of
|
|
an object header. This gives file consistency checking utilities a better chance of reconstructing
|
|
a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This field has a value of 2 indicating version 2 of the object header.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field is a bit field indicating additional information about the object header.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit(s)</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-1</code></td>
|
|
<td>This two bit field determines the size of the <em>Size of Chunk \#0</em> field. The values are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>The <em>Size of Chunk \#0</em> field is 1 byte.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>The <em>Size of Chunk \#0</em> field is 2 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>The <em>Size of Chunk \#0</em> field is 4 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>The <em>Size of Chunk \#0</em> field is 8 bytes.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>If set, attribute creation order is tracked.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>If set, attribute creation order is indexed.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>If set, non-default attribute storage phase change values are stored.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>If set, access, modification, change and birth times are stored.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Access Time</td>
|
|
<td>This 32-bit value represents the number of seconds after the UNIX epoch when the object’s
|
|
raw data was last accessed (in other words, read or written). This field is present if bit 5 of
|
|
<em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Modification Time</td>
|
|
<td>This 32-bit value represents the number of seconds after the UNIX epoch when the object’s
|
|
raw data was last modified (in other words, written). This field is present if bit 5 of
|
|
<em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Change Time</td>
|
|
<td>This 32-bit value represents the number of seconds after the UNIX epoch when the object’s
|
|
metadata was last changed. This field is present if bit 5 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Birth Time</td>
|
|
<td>This 32-bit value represents the number of seconds after the UNIX epoch when the object was
|
|
created. This field is present if bit 5 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Maximum \# of compact attributes</td>
|
|
<td>This is the maximum number of attributes to store in the compact format before switching to the
|
|
indexed format. This field is present if bit 4 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Minimum \# of dense attributes</td>
|
|
<td>This is the minimum number of attributes to store in the indexed format before switching to the
|
|
compact format. This field is present if bit 4 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Chunk \#0</td>
|
|
<td>This unsigned value specifies the number of bytes of header message data following this field,
|
|
within @ref FMT4OhdrChunkDef "chunk \#0" (the messages, and the Gap, if present, but not the
|
|
Checksum field). This value does not include the size of object header continuation blocks
|
|
(chunk \#1 and beyond) for this object elsewhere in the file. The length of this field varies
|
|
depending on bits 0 and 1 of the <em>flags</em> field.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Type</td>
|
|
<td>Same format as version 1 of the object header, described above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Header Message \#n Data</td>
|
|
<td>This value specifies the number of bytes of header message data following the header message
|
|
type, length, flags, and creation order (when present) fields for the current message. The size
|
|
of messages in this version does <em>not</em> include any padding bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Flags</td>
|
|
<td>Same format as version 1 of the object header, described above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Creation Order</td>
|
|
<td>This field stores the order that a message of a given type was created in.<br />
|
|
This field is present if bit 2 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Data</td>
|
|
<td>Same format as version 1 of the object header, described above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Gap</td>
|
|
<td>A gap in @ref FMT4OhdrChunkDef "chunk \#0" is inferred by the end of the messages for the chunk
|
|
before the beginning of this chunk’s Checksum field. Gaps are always smaller than the size
|
|
of an object header message prefix (message type + message size + message flags).<br />
|
|
Gaps are formed when a message (typically an attribute message) in an earlier chunk is deleted
|
|
and a message from a later chunk that does not quite fit into the free space is moved into the
|
|
earlier chunk.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for chunk \#0 only (see @ref FMT4OhdrChunkDef "object header chunks"),
|
|
computed over every byte of this chunk from the <code>OHDR</code> signature through the end of
|
|
this chunk’s Gap field (if present), excluding this Checksum field itself. Each continuation
|
|
chunk, if any, carries its own separate checksum; see
|
|
@ref subsubsec_fmt4_dataobject_hdr_msg_continuation.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The header message types and the message data associated with them compose the critical "meta-data" about
|
|
each object. Some header messages are required for each object while others are optional. Some optional
|
|
header messages may also be repeated several times in the header itself, the requirements and number of
|
|
times allowed in the header will be noted in each header message description below.
|
|
|
|
\subsection subsec_fmt4_dataobject_hdr_msg IV.A.2 Disk Format: Level 2A2 - Data Object Header Shared Message Encoding
|
|
Data object header message data is either stored directly in the data object header for the object or is
|
|
shared between multiple objects in the file. When a message is shared, a flag in the <em>Message Flags</em>
|
|
indicates that the actual <em>Message Data</em> portion of that message is stored in another location
|
|
(such as another data object header, or a heap in the file) and the <em>Message Data</em> field
|
|
contains the information needed to locate the actual information for the message.
|
|
|
|
The format of shared message data is described here:
|
|
<table>
|
|
<caption><strong>Layout: Shared Message (Version 1)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Type</td>
|
|
<td colspan="2">Reserved (zero)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Reserved (zero)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Link Name Offset<sup>L</sup></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Shared Message (Version 1)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number is used when there are changes in the format of a shared object message and is
|
|
described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Version</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Used by the library before version 1.6.1.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Type</td>
|
|
<td>The type of shared message location:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Message stored in another object’s header (a <em>committed</em> message).</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link Name Offset</td>
|
|
<td>This field is skipped over, but not read, by the library’s shared message decoding code. Its
|
|
name is deduced matching the <em>Link Name Offset</em> field of the
|
|
@ref subsec_fmt4_infra_symboltableentry structure, which this shared message format reuses.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>The address of the object header containing the message to be shared.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Shared Message (Version 2)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Type</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Shared Message (Version 2)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number is used when there are changes in the format of a shared object message and is
|
|
described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Version</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Used by the library of version 1.6.1 and after.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Type</td>
|
|
<td>The type of shared message location:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Message stored in another object’s header (a <em>committed</em> message).</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>The address of the object header containing the message to be shared.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Shared Message (Version 3)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Type</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Location <em>(variable size)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Shared Message (Version 3)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number indicates changes in the format of shared object message and is described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Version</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Used by the library of version 1.8 and after. In this version, the <em>Type</em> field can
|
|
indicate that the message is stored in the fractal heap.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Type</td>
|
|
<td>The type of shared message location:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Message is not shared and is not shareable.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Message stored in file’s <em>shared object header message</em> heap
|
|
(a <em>shared</em> message).</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Message stored in another object’s header (a <em>committed</em> message).</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Message stored is not shared, but is shareable.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Location</td>
|
|
<td>This field contains either a <em>@ref FMT4SizeOfOffsetsV0 "Size of Offsets"</em>-bytes address of the object header
|
|
containing the message to be shared, or an 8-byte fractal heap ID for the message in the
|
|
file’s <em>shared object header message</em> heap.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_dataobject_hdr_msg_catalog IV.A.3 Disk Format: Level 2A3 - Data Object Header Messages
|
|
Data object header messages are small pieces of metadata that are stored in the data object header for
|
|
each object in an HDF5 file. Data object header messages provide the metadata required to describe an
|
|
object and its contents, as well as optional pieces of metadata that annotate the meaning or purpose of
|
|
the object.
|
|
|
|
The following is a list of currently defined header messages:
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_nil IV.A.3.a. The NIL Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> NIL</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0000</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The NIL message is used to indicate a message which is to be ignored when reading the header messages
|
|
for a data object. [Possibly one which has been deleted for some reason.]</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> Unspecified</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_simple IV.A.3.b. The Dataspace Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Dataspace</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0001</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies according to the number of dimensions, as described in the following
|
|
table.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Required for dataset objects; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The dataspace message describes the number of dimensions, <em>d</em> (also called the
|
|
“rank”), and the extent of each dimension that the data object has. This message is only used for datasets which have a
|
|
simple, rectilinear, array-like layout; datasets requiring a more complex layout are not yet supported.<br />
|
|
This message may be shared between multiple objects in the file; see @ref subsec_fmt4_dataobject_hdr_msg
|
|
for the shared message encoding.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Dataspace Message - Version 1</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td>Dimensionality</td>
|
|
<td>Flags</td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Reserved</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Dimension \#0 Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dimension \#(d-1) Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dimension \#0 Maximum Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dimension \#(d-1) Maximum Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Permutation Index \#0<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Permutation Index \#(d-1)<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Dataspace Message - Version 1</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version </td>
|
|
<td>This value is used to determine the format of the Dataspace Message. When the format of the
|
|
information in the message is changed, the version number is incremented and can be used to determine
|
|
how the information in the object header is formatted. This document describes version one (1) (there
|
|
was no version zero (0)).</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimensionality</td>
|
|
<td>This value is the number of dimensions, <em>d</em>, that the data object has (also known as the
|
|
“rank”).</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Flags</td>
|
|
<td>This field is used to store flags to indicate the presence of parts of this message. Bit 0 (the least
|
|
significant bit) is used to indicate that maximum dimensions are present. Bit 1 is used to indicate
|
|
that permutation indices are present.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimension \#i Extent (i = 0 to (d-1))</td>
|
|
<td>This value is the current extent of the dimension of the data as stored in the file, in units of
|
|
array elements (not bytes). Dimension \#0 is the slowest-changing axis and Dimension \#(d-1) is
|
|
the fastest-changing axis.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimension \#i Maximum Extent (i = 0 to (d-1))</td>
|
|
<td>This value is the maximum extent of the dimension of the data as stored in the file. This value may be
|
|
the special “@ref FMT4UnlimitedDim "unlimited extent"” which indicates that the data
|
|
may expand along this dimension indefinitely. If these values are not stored, the maximum extent of
|
|
each dimension is assumed to be the dimension’s current extent.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Permutation Index \#i (i = 0 to (d-1))</td>
|
|
<td>This value is the index permutation used to map each dimension from the canonical representation to an
|
|
alternate axis for each dimension. If these values are not stored, Dimension \#0 is the slowest changing
|
|
dimension and Dimension \#(d-1) is the fastest changing dimension.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
Version 2 of the dataspace message dropped the optional permutation index value support, as it was never
|
|
implemented in the HDF5 Library:
|
|
<table>
|
|
<caption align="top"><strong>Layout: Dataspace Message - Version 2</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td>Dimensionality</td>
|
|
<td>Flags</td>
|
|
<td>Type</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Dimension \#0 Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dimension \#(d-1) Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dimension \#0 Maximum Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dimension \#(d-1) Maximum Extent<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Dataspace Message - Version 2</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version </td>
|
|
<td>This value is used to determine the format of the Dataspace Message. This field should be ‘2’
|
|
for version 2 format messages.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimensionality</td>
|
|
<td>This value is the number of dimensions, <em>d</em>, that the data object has (also known as the
|
|
“rank”).</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Flags</td>
|
|
<td>This field is used to store flags to indicate the presence of parts of this message. Bit 0 (the least
|
|
significant bit) is used to indicate that maximum dimensions are present.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Type</td>
|
|
<td>indicates the type of the dataspace:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>A <em>scalar</em> dataspace; in other words, a dataspace with a single, dimensionless element.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>A <em>simple</em> dataspace; in other words, a dataspace with a rank > 0 and an appropriate number
|
|
of dimensions.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>A <em>null</em> dataspace; in other words, a dataspace with no elements.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimension \#i Extent (i = 0 to (d-1))</td>
|
|
<td>This value is the current extent of the dimension of the data as stored in the file, in units of
|
|
array elements (not bytes). Dimension \#0 is the slowest-changing axis and Dimension \#(d-1) is
|
|
the fastest-changing axis.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimension \#i Maximum Extent (i = 0 to (d-1))</td>
|
|
<td>This value is the maximum extent of the dimension of the data as stored in the file. This value may be
|
|
the special “@ref FMT4UnlimitedDim "unlimited extent"” which indicates that the data
|
|
may expand along this dimension indefinitely. If these values are not stored, the maximum extent of
|
|
each dimension is assumed to be the dimension’s current extent.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_linkinfo IV.A.3.c. The Link Info Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Link Info</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x002</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Required for “new style” groups; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The link info message tracks variable information about the current state of the links for a
|
|
“new style” group’s behavior. Variable information will be stored in this
|
|
message and constant information will be stored in the @ref
|
|
subsubsec_fmt4_dataobject_hdr_msg_groupinfo message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Link Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Maximum Creation Index <em>(8 bytes, optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Fractal Heap Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of v2 B-tree for Name Index<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address of v2 B-tree for Creation Order Index<sup>O</sup> <em>(optional)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Link Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field determines various optional aspects of the link info message:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, creation order for the links is tracked.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, creation order for the links is indexed.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-7</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Maximum Creation Index</td>
|
|
<td>This 64-bit value is the maximum creation order index value stored for a link in this group.<br />
|
|
This field is present if bit 0 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Fractal Heap Address</td>
|
|
<td>This is the address of the fractal heap to store dense links. Each link stored in the fractal heap
|
|
is stored as a @ref subsubsec_fmt4_dataobject_hdr_msg_link.<br />
|
|
If there are no links in the group, or the group’s links are stored “compactly”
|
|
(as object header messages), this value will be the @ref FMT4UndefinedAddress "undefined address".</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address of v2 B-tree for Name Index</td>
|
|
<td>This is the address of the version 2 B-tree to index names of links.<br />
|
|
If there are no links in the group, or the group’s links are stored “compactly”
|
|
(as object header messages), this value will be the @ref FMT4UndefinedAddress "undefined address".</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address of v2 B-tree for Creation Order Index</td>
|
|
<td>This is the address of the version 2 B-tree to index creation order of links.<br />
|
|
If there are no links in the group, or the group’s links are stored “compactly”
|
|
(as object header messages), this value will be the @ref FMT4UndefinedAddress "undefined address".<br />
|
|
This field exists if bit 1 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_dtmessage IV.A.3.d. The Datatype Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Datatype</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0003</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Variable</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Required for dataset or committed datatype (formerly named datatype)
|
|
objects; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The datatype message defines the datatype for each element of a dataset or a common datatype for
|
|
sharing between multiple datasets. A datatype can describe an atomic type like a fixed- or
|
|
floating-point type or more complex types like a C struct (compound datatype), array (array datatype)
|
|
or C++ vector (variable-length datatype).<br />
|
|
Datatype messages that are part of a dataset object do not describe how elements are related to one
|
|
another; the dataspace message is used for that purpose.<br />
|
|
Datatype messages that are part of a committed datatype (formerly named datatype) message describe a
|
|
common datatype that can be shared by multiple datasets in the file. See @ref
|
|
subsec_fmt4_dataobject_hdr_msg for the shared message encoding.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Datatype Message</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Class and Version</td>
|
|
<td>Class Bit Field, Bits 0-7</td>
|
|
<td>Class Bit Field, Bits 8-15</td>
|
|
<td>Class Bit Field, Bits 16-23</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Size</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br /><br />Properties<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Datatype Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Class and Version</td>
|
|
<td>The version of the datatype message and the datatype’s class information are packed together in
|
|
this field. The version number is packed in the top 4 bits of the field and the class is contained
|
|
in the bottom 4 bits.<br />
|
|
The version number information is used for changes in the format of the datatype message and is
|
|
described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Used by early versions of the library to encode compound datatypes with explicit array fields.
|
|
See the compound datatype description below for further details.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Used when an array datatype needs to be encoded.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Used when a VAX byte-ordered type needs to be encoded. Packs various other datatype classes more
|
|
efficiently also.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>Used to encode the revised reference datatype.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>Used when a complex number datatype needs to be encoded.</td>
|
|
</tr>
|
|
</table><br />
|
|
The class of the datatype determines the format for the class bit field and properties portion of the
|
|
datatype message, which are described below. The following classes are currently defined:
|
|
<table width=100%>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>\ref FMT4ClassFixedPoint "Fixed-Point"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>\ref FMT4ClassFloatingPoint "Floating-Point"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>\ref FMT4ClassTime "Time"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>\ref FMT4ClassString "String"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>\ref FMT4ClassBitField "Bit field"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>\ref FMT4ClassOpaque "Opaque"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6</code></td>
|
|
<td>\ref FMT4ClassCompound "Compound"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>7</code></td>
|
|
<td>\ref FMT4ClassReference "Reference"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>8</code></td>
|
|
<td>\ref FMT4ClassEnum "Enumerated"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>9</code></td>
|
|
<td>\ref FMT4ClassVarLen "Variable-Length"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>10</code></td>
|
|
<td>\ref FMT4ClassArray "Array"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>11</code></td>
|
|
<td>\ref FMT4ClassComplex "Complex"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Class Bit Fields</td>
|
|
<td>The information in these bit fields is specific to each datatype class and is described below.
|
|
All bits not defined for a datatype class are set to zero.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size</td>
|
|
<td>The size of a datatype element in bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Properties</td>
|
|
<td>This variable-sized sequence of bytes encodes information specific to each datatype class and is
|
|
described for each class below. If there is no property information specified for a datatype class,
|
|
the size of this field is zero bytes.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassFixedPoint <h4>Class specific information for Fixed-Point Numbers (Class 0):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Fixed-point Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0</td>
|
|
<td><strong>Byte Order.</strong> If zero, byte order is little-endian; otherwise, byte order is big endian.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>1, 2</td>
|
|
<td><strong>Padding type.</strong> Bit 1 is the lo_pad type and bit 2 is the hi_pad type. If a datum has
|
|
unused bits at either end, then the lo_pad or hi_pad bit is copied to those locations.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>3</td>
|
|
<td><strong>Signed.</strong> If this bit is set then the fixed-point number is in 2’s complement form.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>4-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Fixed-Point Property Descriptions</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Bit Offset</td>
|
|
<td colspan="2">Bit Precision</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Fixed-Point Property Descriptions</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Bit Offset</td>
|
|
<td>The bit offset of the first significant bit of the fixed-point value within the datatype. The
|
|
bit offset specifies the number of bits “to the right of” the value (which are
|
|
set to the lo_pad bit value).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Bit Precision</td>
|
|
<td>The number of bits of precision of the fixed-point value within the datatype. This value,
|
|
combined with the datatype element’s size and the Bit Offset field specifies the number
|
|
of bits “to the left of” the value (which are set to the hi_pad bit value).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassFloatingPoint <h4>Class specific information for Floating-Point Numbers (Class 1):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Floating-Point Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0</td>
|
|
<td><strong>Byte Order.</strong> These two non-contiguous bits specify the “endianness” of
|
|
the bytes in the datatype element.
|
|
<table>
|
|
<tr>
|
|
<th width="10%" align="center">Bit 6</th>
|
|
<th width="10%" align="center">Bit 0</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Byte order is little-endian</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Byte order is big-endian</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Byte order is VAX-endian</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>1, 2, 3</td>
|
|
<td><strong>Padding type.</strong> Bit 1 is the low bits pad type, bit 2 is the high bits pad type, and bit
|
|
3 is the internal bits pad type. If a datum has unused bits at either end or between the sign bit, exponent,
|
|
or mantissa, then the value of bit 1, 2, or 3 is copied to those locations.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>4-5</td>
|
|
<td><strong>Mantissa Normalization.</strong> This 2-bit bit field specifies how the most significant bit of
|
|
the mantissa is managed.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>No normalization</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>The most significant bit of the mantissa is always set (except for 0.0).</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>The most significant bit of the mantissa is not stored, but is implied to be set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>6-7</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>8-15</td>
|
|
<td><strong>Sign Location.</strong> This is the bit position of the sign bit. Bits are numbered with the least
|
|
significant bit zero.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>16-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Floating-Point Property Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Bit Offset</td>
|
|
<td colspan="2">Bit Precision</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Exponent Location</td>
|
|
<td>Exponent Size</td>
|
|
<td>Mantissa Location</td>
|
|
<td>Mantissa Size</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Exponent Bias</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Floating-Point Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Property Description</th>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>Bit Offset</td>
|
|
<td>The bit offset of the first significant bit of the floating-point value within the datatype. The
|
|
bit offset specifies the number of bits “to the right of” the value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Bit Precision</td>
|
|
<td>The number of bits of precision of the floating-point value within the datatype.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Exponent Location</td>
|
|
<td>The bit position of the exponent field. Bits are numbered with the least significant
|
|
bit number zero.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Exponent Size</td>
|
|
<td>The size of the exponent field in bits.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Mantissa Location</td>
|
|
<td>The bit position of the mantissa field. Bits are numbered with the least significant bit number
|
|
zero.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Mantissa Size</td>
|
|
<td>The size of the mantissa field in bits.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Exponent Bias</td>
|
|
<td>The bias of the exponent field.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassTime <h4>Class specific information for Time (Class 2):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Time Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0</td>
|
|
<td><strong>Byte Order.</strong> If zero, byte order is little-endian; otherwise, byte order is big endian.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>1-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
|
|
<table>
|
|
<caption><strong>Layout: Time Property Description</strong></caption>
|
|
<tr>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Bit Precision</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
|
|
<table>
|
|
<caption><strong>Fields: Time Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Bit Precision</td>
|
|
<td>The number of bits of precision of the time value.</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
|
|
\anchor FMT4ClassString <h4>Class specific information for Strings (Class 3):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: String Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0-3</td>
|
|
<td><strong>Padding type.</strong> This four-bit value determines the type of padding to use for the
|
|
string. The values are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td><code>0</code></td>
|
|
<td>Null Terminate: A zero byte marks the end of the string and is guaranteed to be present after
|
|
converting a long string to a short string. When converting a short string to a long string the
|
|
value is padded with additional null characters as necessary.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>1</code></td>
|
|
<td>Null Pad: Null characters are added to the end of the value during conversions from short values
|
|
to long values but conversion in the opposite direction simply truncates the value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>2</code></td>
|
|
<td>Space Pad: Space characters are added to the end of the value during conversions from short values
|
|
to long values but conversion in the opposite direction simply truncates the value. This is the
|
|
Fortran representation of the string.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><code>3-15</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>4-7</td>
|
|
<td><strong>Character Set.</strong> The character set used to encode the string.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>ASCII character set encoding</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>UTF-8 character set encoding</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-15</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>8-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
There are no properties defined for the string class.
|
|
|
|
\anchor FMT4ClassBitField <h4>Class specific information for Bitfields (Class 4):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Bitfield Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0</td>
|
|
<td><strong>Byte Order.</strong> If zero, byte order is little-endian; otherwise, byte order is big endian.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>1, 2</td>
|
|
<td><strong>Padding type.</strong> Bit 1 is the lo_pad type and bit 2 is the hi_pad type. If a datum has
|
|
unused bits at either end, then the lo_pad or hi_pad bit is copied to those locations.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>3-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Bit Field Property Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Bit Offset</td>
|
|
<td colspan="2">Bit Precision</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Bit Field Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Bit Offset</td>
|
|
<td>The bit offset of the first significant bit of the bitfield within the datatype. The bit
|
|
offset specifies the number of bits “to the right of” the value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Bit Precision</td>
|
|
<td>The number of bits of precision of the bit field within the datatype.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassOpaque <h4>Class specific information for Opaque (Class 5):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Opaque Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr>
|
|
<td>0-7</td>
|
|
<td>Length of ASCII tag in bytes.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>8-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Opaque Property Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />ASCII Tag<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Opaque Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>ASCII Tag</td>
|
|
<td>This NUL-terminated string provides a description for the opaque type. It is NUL-padded to a
|
|
multiple of 8 bytes.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassCompound <h4>Class specific information for Compound Types (Class 6):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Compound Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0-15</td>
|
|
<td><strong>Number of Members.</strong> This field contains the number of members defined for the
|
|
compound datatype. The member definitions are listed in the Properties field of the data type
|
|
message.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>16-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The Properties field of a compound datatype is a list of the member definitions of the compound datatype.
|
|
The member definitions appear one after another with no intervening bytes. The member types are described
|
|
with a (recursively) encoded datatype message.
|
|
|
|
Note that the property descriptions are different for different versions of the datatype version. Additionally
|
|
note that the version 0 properties are deprecated and has been replaced with later encodings in
|
|
versions of the HDF5 library from the 1.4 release onward.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Compound Properties Description for Datatype Version 1</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br /><br />Name<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Byte Offset of Member</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Dimensionality</td>
|
|
<td colspan="3">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Dimension Permutation</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Dimension \#0 Extent (required)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Dimension \#1 Extent (required)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Dimension \#2 Extent (required)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Dimension \#3 Extent (required)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Member Type Message<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Compound Properties Description for Datatype Version 1</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Name</td>
|
|
<td>This NUL-terminated string is the name of the member. It is NUL-padded to a
|
|
multiple of 8 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Byte Offset of Member</td>
|
|
<td>This is the byte offset of the member within the datatype.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td>If set to zero, this field indicates a scalar member. If set to a value greater than zero,
|
|
<em>d</em>, this field indicates that the member is an array of <em>d</em> dimensions, whose
|
|
extents are given by Dimension \#0 Extent through Dimension \#(d−1) Extent; the
|
|
remaining Dimension Extent fields, if any, are unused.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension Permutation</td>
|
|
<td>This field was intended to allow an array field to have its dimensions permuted, but this was
|
|
never implemented. This field should always be set to zero.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension \#i Extent (i = 0 to 3)</td>
|
|
<td>This field is the extent of a dimension of the array field, in units of array elements (not
|
|
bytes), as stored in the file. Dimension \#0 is the slowest-changing axis. Exactly 4 Dimension
|
|
Extent fields are always present; only the first <em>d</em> (Dimension \#0 Extent through
|
|
Dimension \#(d−1) Extent) are meaningful, and any remaining fields are unused.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Member Type Message</td>
|
|
<td>This field is a datatype message describing the datatype of the member.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Compound Properties Description for Datatype Version 2</strong></caption>
|
|
<tr>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Name<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Byte Offset of Member</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Member Type Message<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Compound Properties Description for Datatype Version 2</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Name</td>
|
|
<td>This NUL-terminated string is the name of the member. It is NUL-padded to a multiple
|
|
of 8 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Byte Offset of Member</td>
|
|
<td>This is the byte offset of the member within the datatype.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Member Type Message</td>
|
|
<td>This field is a datatype message describing the datatype of the member.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Compound Properties Description for Datatype Version 3</strong></caption>
|
|
<tr>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Name<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Byte Offset of Member</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Member Type Message<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Compound Properties Description for Datatype Version 3</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Name</td>
|
|
<td>This NUL-terminated string is the name of the member. It is <em>not</em> NUL-padded to a
|
|
multiple of 8 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Byte Offset of Member</td>
|
|
<td>This is the byte offset of the member within the datatype. The field size is the minimum number of bytes
|
|
necessary, based on the size of the datatype element. For example, a datatype element size of less than
|
|
256 bytes uses a 1 byte length, a datatype element size of 256-65535 bytes uses a 2 byte length, and
|
|
so on.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Member Type Message</td>
|
|
<td>This field is a datatype message describing the datatype of the member.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassReference <h4>Class specific information for Reference (Class 7):</h4><br />
|
|
Note that for region references, the stored data is a \ref FMT4GlobalHeapID "Global Heap ID" pointing to information
|
|
about the region stored in the global heap.
|
|
|
|
<table>
|
|
<caption><strong>Bits: Reference Bit Field Description for Datatype Version < 4</strong></caption>
|
|
<tr>
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr>
|
|
<td>0-3</td>
|
|
<td><strong>Type.</strong> This four-bit value contains the reference types which are supported
|
|
for backward compatibility. The values defined are:
|
|
<table width=100%>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Object Reference (#H5R_OBJECT1): A reference to another object in this HDF5 file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Dataset Region Reference (#H5R_DATASET_REGION1): A reference to a region within a dataset
|
|
in this HDF5 file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-15</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>4-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Bits: Reference Bit Field Description for Datatype Version 4</strong></caption>
|
|
<tr>
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr>
|
|
<td>0-3</td>
|
|
<td><strong>Type.</strong> This four-bit value contains the revised generic reference type. The value
|
|
defined is:
|
|
<table width=100%>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>It covers all three revised reference types: Object Reference (#H5R_OBJECT2), Dataset Region Reference (#H5R_DATASET_REGION2), and Attribute Reference (#H5R_ATTR).
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3-15</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>4-7</td>
|
|
<td><strong>Version.</strong> This four-bit value contains the version for encoding the revised
|
|
reference types. The values defined are:
|
|
<table width=100%>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Unused</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>The version for encoding the revised reference types: Object Reference (2),
|
|
Dataset Region Reference (3) and Attribute Reference (4).</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-15</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>8-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
There are no properties defined for the reference class.
|
|
|
|
\anchor FMT4ClassEnum <h4>Class specific information for Enumeration (Class 8):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Enumeration Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0-15</td>
|
|
<td><strong>Number of Members.</strong> The number of name/value pairs defined for the enumeration type.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>16-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Enumeration Property Description for Datatype Versions 1 and 2</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Base Type<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Names<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Values<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Enumeration Property Description for Datatype Versions 1 and 2</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr align="left" valign=top>
|
|
<td valign=top>Base Type</td>
|
|
<td valign=top>Each enumeration type is based on some parent type, usually an integer. The information
|
|
for that parent type is described recursively by this field.</td>
|
|
</tr>
|
|
<tr align="left" valign=top>
|
|
<td valign=top>Names</td>
|
|
<td valign=top>The name for each name/value pair. Each name is stored as a null terminated string
|
|
in a multiple of eight bytes. The format does not constrain the character encoding of the
|
|
string beyond disallowing a NUL byte within the name itself, since the format always appends
|
|
one as a terminator; the library has historically documented it as ASCII, but the reference
|
|
implementation does not reject other encodings, such as UTF-8. The names are in no particular
|
|
order.</td>
|
|
</tr>
|
|
<tr align="left" valign=top>
|
|
<td valign=top>Values</td>
|
|
<td valign=top>The list of values in the same order as the names. The values are packed (no inter-value
|
|
padding) and the size of each value is determined by the parent type.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Enumeration Property Description for Datatype Versions 3</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Base Type<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Names<br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Values<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Enumeration Property Description for Datatype Versions 3</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr align="left" valign=top>
|
|
<td valign=top>Base Type</td>
|
|
<td valign=top>Each enumeration type is based on some parent type, usually an integer. The information
|
|
for that parent type is described recursively by this field.</td>
|
|
</tr>
|
|
<tr align="left" valign=top>
|
|
<td valign=top>Names</td>
|
|
<td valign=top>The name for each name/value pair. Each name is stored as a null terminated string,
|
|
<em>not</em> padded to a multiple of eight bytes. The format does not constrain the character
|
|
encoding of the string beyond disallowing a NUL byte within the name itself, since the format
|
|
always appends one as a terminator; the library has historically documented it as ASCII, but
|
|
the reference implementation does not reject other encodings, such as UTF-8. The names are in
|
|
no particular order.</td>
|
|
</tr>
|
|
<tr align="left" valign=top>
|
|
<td valign=top>Values</td>
|
|
<td valign=top>The list of values in the same order as the names. The values are packed (no inter-value
|
|
padding) and the size of each value is determined by the parent type.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassVarLen <h4>Class specific information for Variable-Length (Class 9):</h4>
|
|
<table>
|
|
<caption align="top"><strong>Bits: Variable-Length Bit Field Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>0-3</td>
|
|
<td><strong>Type.</strong> This four-bit value contains the type of variable-length datatype described.
|
|
The values defined are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Sequence: A variable-length sequence of any datatype. Variable-length sequences do not
|
|
have padding or character set information.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>String: A variable-length sequence of characters. Variable-length strings have padding and
|
|
character set information.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-15</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>4-7</td>
|
|
<td><strong>Padding type.</strong> (variable-length string only). This four-bit value determines the
|
|
type of padding used for variable-length strings. The values are the same as for the string padding
|
|
type, as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>0 Null terminate: A zero byte marks the end of a string and is guaranteed to be present after
|
|
converting a long string to a short string. When converting a short string to a long string,
|
|
the value is padded with additional null characters as necessary.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Null pad: Null characters are added to the end of the value during conversion from a short string to
|
|
a longer string. Conversion from a long string to a shorter string simply truncates the value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Space pad: Space characters are added to the end of the value during conversion from a short string
|
|
to a longer string. Conversion from a long string to a shorter string simply truncates the value.
|
|
This is the Fortran representation of the string.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3-15</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table>
|
|
This value is set to zero for variable-length sequences.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>8-11</td>
|
|
<td><strong>Character Set.</strong> (variable-length string only) This four-bit value specifies the
|
|
character set to be used for encoding the string:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>ASCII character set encoding.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>UTF-8 character set encoding.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-15</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table>
|
|
This value is set to zero for variable-length sequences.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>12-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Variable-Length Property Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Parent Type Message<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Variable-Length Property Description</strong></caption>
|
|
<tr align=center>
|
|
<th width="10%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr align=center>
|
|
<td>Parent Type</td>
|
|
<td>Each variable-length type is based on some parent type. This field contains the datatype
|
|
message describing that parent type. In the case of nested variable-length types, this parent
|
|
datatype message will recursively contain all parent datatype messages. Variable-length strings
|
|
are considered to have the parent type #H5T_NATIVE_UCHAR.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassArray <h4>Class specific information for Array (Class 10):</h4>
|
|
|
|
There are no bit fields defined for the array class.
|
|
|
|
Note that the dimension information defined in the property for this datatype class is independent of
|
|
dataspace information for a dataset. The dimension information here describes the dimensionality of the
|
|
information within a data element (or a component of an element, if the array datatype is nested within
|
|
another datatype) and the dataspace for a dataset describes the location of the elements in a dataset.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Array Property Description for Datatype Version 2</strong></caption>
|
|
<tr>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td colspan="3">Reserved (zero)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#0 Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#(d-1) Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Permutation Index \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Permutation Index \#(d-1)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Base Type<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Array Property Description for Datatype Version 2</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td>This value is the number of dimensions, <em>d</em>, that the array has.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension \#i Extent (i = 0 to (d-1))</td>
|
|
<td>This value is the extent of the dimension of the array as stored in the file, in units of array
|
|
elements (not bytes). Dimension \#0 is the slowest-changing axis and Dimension \#(d-1) is the
|
|
fastest-changing axis.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Permutation Index \#i (i = 0 to (d-1))</td>
|
|
<td>This value is the index permutation used to map each dimension from the canonical representation
|
|
to an alternate axis for each dimension. Currently, dimension permutations are not supported and
|
|
these indices should be set to the index position (i.e. Permutation Index \#0 should be set to 0,
|
|
Permutation Index \#1 should be set to 1, and so on.)</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Base Type</td>
|
|
<td>Each array type is based on some parent type. The information for that parent type is described
|
|
recursively by this field.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Array Property Description for Datatype Version 3</strong></caption>
|
|
<tr>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#0 Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#(d-1) Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Base Type<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Array Property Description for Datatype Version 3</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td>This value is the number of dimensions, <em>d</em>, that the array has.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension \#i Extent (i = 0 to (d-1))</td>
|
|
<td>This value is the extent of the dimension of the array as stored in the file, in units of array
|
|
elements (not bytes). Dimension \#0 is the slowest-changing axis and Dimension \#(d-1) is the
|
|
fastest-changing axis.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Base Type</td>
|
|
<td>Each array type is based on some parent type. The information for that parent type is described
|
|
recursively by this field.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ClassComplex <h4>Class specific information for the Complex class (Class 11):</h4>
|
|
<table>
|
|
<caption><strong>Bits: Complex Bit Field Description</strong></caption>
|
|
<tr>
|
|
<th width="10%">Bits</th>
|
|
<th>Meaning</th>
|
|
</tr>
|
|
<tr>
|
|
<td>0</td>
|
|
<td><strong>Homogeneous.</strong> If zero, each part of the complex number
|
|
datatype is a different floating point datatype (heterogeneous).
|
|
Otherwise, each part of the complex number datatype is the same
|
|
floating point datatype (homogeneous). Currently, only homogeneous
|
|
complex number datatypes are supported.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>1,2</td>
|
|
<td><b>Complex number form.</b> This two-bit value contains the type of complex number datatype
|
|
described. The values defined are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Rectangular</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Polar</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Exponential</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table>
|
|
Currently, only rectangular complex number datatypes are supported.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>3-23</td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Complex Property Description</strong></caption>
|
|
<tr>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th width="25%">Byte</th>
|
|
<th>Byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Parent Type Message<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Complex Property Description</strong></caption>
|
|
<tr>
|
|
<th width="10%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Parent Type Message</td>
|
|
<td>Each complex number type is based on a parent floating point type. This field contains the
|
|
datatype message describing that parent type.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_ofvmessage IV.A.3.e. Data Storage - Fill Value (Old) Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Fill Value (old)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0004</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The fill value message stores a single data value which is returned to the application when an
|
|
uninitialized data element is read from a dataset. The fill value is interpreted with the same
|
|
datatype as the dataset. If no fill value message is present then a fill value of all zero bytes
|
|
is assumed.<br />
|
|
This fill value message is deprecated in favor of the “new” fill value message (Message
|
|
Type 0x0005) and is only written to the file for forward compatibility with versions of the HDF5
|
|
Library before the 1.6.0 version. Additionally, it only appears for datasets with a user-defined
|
|
fill value (as opposed to the library default fill value or an explicitly set “undefined”
|
|
fill value).<br />
|
|
This message may be shared between multiple objects in the file; see @ref subsec_fmt4_dataobject_hdr_msg
|
|
for the shared message encoding.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Fill Value Message (Old)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Size (4 bytes)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Fill Value <em>(optional, variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption align="top"><strong>Fields: Fill Value Message (Old)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Size</td>
|
|
<td>This is the size of the Fill Value field in bytes.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Fill Value</td>
|
|
<td>The fill value. The bytes of the fill value are interpreted using the same datatype as for the dataset.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_fvmessage IV.A.3.f. The Data Storage - Fill Value Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Fill Value</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0005</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Required for dataset objects; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The fill value message stores a single data value which is returned to the application when an
|
|
uninitialized data element is read from a dataset. The fill value is interpreted with the same
|
|
datatype as the dataset.<br />
|
|
This message may be shared between multiple objects in the file; see @ref subsec_fmt4_dataobject_hdr_msg
|
|
for the shared message encoding.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Fill Value Message - Versions 1 & 2</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Space Allocation Time</td>
|
|
<td>Fill Value Write Time</td>
|
|
<td>Fill Value Defined</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Size <em>(optional)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Fill Value <em>(optional, variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Fill Value Message - Versions 1 & 2</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of the fill value message and
|
|
is described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Initial version of this message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>In this version, the Size and Fill Value fields are only present if the Fill Value Defined
|
|
field is set to 1.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>This version packs the other fields in the message more efficiently than version 2.</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Space Allocation Time</td>
|
|
<td>When the storage space for the dataset’s raw data will be allocated. The allowed values are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Not used</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Early allocation. Storage space for the entire dataset should be allocated in the file
|
|
when the dataset is created.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Late allocation. Storage space for the entire dataset should not be allocated until the
|
|
dataset is written to.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Incremental allocation. Storage space for the dataset should not be allocated until the
|
|
portion of the dataset is written to. This is currently used in conjunction with chunked
|
|
data storage for datasets.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Fill Value Write Time</td>
|
|
<td>At the time that storage space for the dataset’s raw data is allocated, this value indicates
|
|
whether the fill value should be written to the raw data storage elements. The allowed values are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>On allocation. The fill value is always written to the raw data storage when the storage
|
|
space is allocated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Never. The fill value should never be written to the raw data storage.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Fill value written if set by user. The fill value will be written to the raw data storage
|
|
when the storage space is allocated only if the user explicitly set the fill value. If the
|
|
fill value is the library default or is undefined, it will not be written to the raw data storage.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Fill Value Defined</td>
|
|
<td>This value indicates if a fill value is defined for this dataset. If this value is 0, the fill
|
|
value is undefined. If this value is 1, a fill value is defined for this dataset. For version 2
|
|
or later of the fill value message, this value controls the presence of the Size and Fill field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Size</td>
|
|
<td>This is the size of the Fill Value field in bytes. This field is not present if the Version
|
|
field is greater than 1 and the Fill Value Defined field is set to 0.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Fill Value</td>
|
|
<td>The fill value. The bytes of the fill value are interpreted using the same datatype as for the
|
|
dataset. This field is not present if the Version field is greater than 1 and the Fill Value
|
|
Defined field is set to 0.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Fill Value Message - Versions 3</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td>Fill Value Write Time</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Size <em>(optional)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Fill Value <em>(optional, variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Fill Value Message - Versions 3</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of the fill value message and
|
|
is described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Initial version of this message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>In this version, the Size and Fill Value fields are only present if the Fill Value Defined
|
|
field is set to 1.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>This version packs the other fields in the message more efficiently than version 2.</td>
|
|
</tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>When the storage space for the dataset’s raw data will be allocated. The allowed values are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Bits</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-1</code></td>
|
|
<td>Space Allocation Time, with the same values as versions 1 and 2 of the message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-3</code></td>
|
|
<td>Fill Value Write Time, with the same values as versions 1 and 2 of the message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>Fill Value Undefined, indicating that the fill value has been marked as “undefined”
|
|
for this dataset. Bits 4 and 5 cannot both be set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>Fill Value Defined, with the same values as versions 1 and 2 of the message. Bits 4 and 5
|
|
cannot both be set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6-7</code></td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Size</td>
|
|
<td>This is the size of the Fill Value field in bytes. This field is not present if the Version
|
|
field is greater than 1 and the Fill Value Defined flag is set to 0.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Fill Value</td>
|
|
<td>The fill value. The bytes of the fill value are interpreted using the same datatype as for the
|
|
dataset. This field is not present if the Version field is greater than 1 and the Fill Value
|
|
Defined flag is set to 0.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_link IV.A.3.g. The Link Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Link</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0006</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message encodes the information for a link in a group’s object header, when the group is
|
|
storing its links “compactly”, or in the group’s fractal heap, when the group is
|
|
storing its links “densely”.<br />
|
|
A group is storing its links compactly when the fractal heap address in the
|
|
<em>@ref subsubsec_fmt4_dataobject_hdr_msg_linkinfo</em> is set to the
|
|
@ref FMT4UndefinedAddress "undefined address" value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Link Message</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td>Link type <em>(optional)</em></td>
|
|
<td bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Creation Order <em>(8 bytes, optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link Name Character Set <em>(optional)</em></td>
|
|
<td>Length of Link Name <em>(variable size)</em></td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Link Name <em>(variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Link Information <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Link Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 1.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field contains information about the link and controls the presence of other fields below.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bits</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0-1</code></td>
|
|
<td>Determines the size of the <em>Length of Link Name</em> field.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>The size of the <em>Length of Link Name</em> field is 1 byte.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>The size of the <em>Length of Link Name</em> field is 2 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>The size of the <em>Length of Link Name</em> field is 4 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>The size of the <em>Length of Link Name</em> field is 8 bytes.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Creation Order Field Present: if set, the <em>Creation Order</em> field is present. If
|
|
not set, creation order information is not stored for links in this group.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Link Type Field Present: if set, the link is not a hard link and the <em>Link Type</em>
|
|
field is present. If not set, the link is a hard link.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>Link Name Character Set Field Present: if set, the link name is not represented with the
|
|
ASCII character set and the <em>Link Name Character Set</em> field is present. If not set,
|
|
the link name is represented with the ASCII character set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5-7</code></td>
|
|
<td>Reserved (zero).</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link type</td>
|
|
<td>This is the link class type and can be one of the following values:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>A hard link (should never be stored in the file)</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>A soft link.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-63</code></td>
|
|
<td>Reserved for future HDF5 internal use.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>64</code></td>
|
|
<td>An external link.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>65-255</code></td>
|
|
<td>Reserved, but available for user-defined link types.</td>
|
|
</tr>
|
|
</table>
|
|
This field is present if bit 3 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Creation Order</td>
|
|
<td>This 64-bit value is an index of the link’s creation time within the group. Values start at
|
|
0 when the group is created an increment by one for each link added to the group. Removing a link
|
|
from a group does not change existing links’ creation order field.<br />
|
|
This field is present if bit 2 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link Name Character Set</td>
|
|
<td>This is the character set for encoding the link’s name:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>ASCII character set encoding (this should never be stored in the file)</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>UTF-8 character set encoding</td>
|
|
</tr>
|
|
</table>
|
|
This field is present if bit 4 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length of link name</td>
|
|
<td>This is the length of the link’s name. The size of this field depends on bits 0 and 1 of <em>Flags</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link name</td>
|
|
<td>This is the name of the link, non-NULL terminated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link information</td>
|
|
<td>The format of this field depends on the <em>link type</em>.<br />
|
|
For <b>hard</b> links, the field is formatted as follows:
|
|
<table>
|
|
<tr>
|
|
<td width="20%"><i>@ref FMT4SizeOfOffsetsV0 "Size of Offsets"</i> bytes:</td>
|
|
<td>The address of the object header for the object that the link points to.</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
For <b>soft</b> links, the field is formatted as follows:
|
|
<table>
|
|
<tr>
|
|
<td width="20%">Bytes 1-2:</td>
|
|
<td>Length of soft link value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><em>Length of soft link value</em> bytes:</td>
|
|
<td>A non-NULL-terminated string storing the value of the soft link.</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
For <b>external</b> links, the field is formatted as follows:
|
|
<table>
|
|
<tr>
|
|
<td width="20%">Bytes 1-2:</td>
|
|
<td>Length of external link value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><em>Length of external link value</em> bytes:</td>
|
|
<td>The first byte contains the version number in the upper 4 bits and flags in the lower 4 bits
|
|
for the external link. Both version and flags are defined to be zero in this document. The
|
|
remaining bytes consist of two NULL-terminated strings, with no padding between them. The first
|
|
string is the name of the HDF5 file containing the object linked to and the second string is the
|
|
full path to the object linked to, within the HDF5 file’s group hierarchy.</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
For <b>user-defined</b> links, the field is formatted as follows:
|
|
<table>
|
|
<tr>
|
|
<td width="20%">Bytes 1-2:</td>
|
|
<td>Length of user-defined data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><em>Length of user-defined link value</em> bytes:</td>
|
|
<td>The data supplied for the user-defined link type.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_external IV.A.3.h. The Data Storage - External Data Files Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> External Data Files</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0007</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The external data storage message indicates that the data for an object is stored outside the HDF5
|
|
file. The filename of the object is stored as a Universal Resource Location (URL) of the actual
|
|
filename containing the data. An external file list record also contains the byte offset of the
|
|
start of the data within the file and the amount of space reserved in the file for that data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: External File List Message</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td colspan="3">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Allocated Slots</td>
|
|
<td colspan="2">Used Slots</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Heap Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Slot Definitions...<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: External File List Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of External Data Storage Message
|
|
and is described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>The current version used by the library.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Allocated Slots</td>
|
|
<td>The total number of slots allocated in the message. Its value must be at least as large as the value
|
|
contained in the Used Slots field. (The current library simply uses the number of Used Slots for this
|
|
message)</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Used Slots</td>
|
|
<td>The number of initial slots which contain valid information.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Heap Address</td>
|
|
<td>This is the address of a local name heap which contains the names for the external files. (The local
|
|
heap information can be found in @ref subsec_fmt4_infra_localheap in this document). The name at offset
|
|
zero in the heap is always the empty string.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Slot Definitions</td>
|
|
<td>The slot definitions are stored in order according to the array addresses they represent.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: External File List Slot</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Name Offset in Local Heap<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />File Offset in External Data File<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Data Size in External File<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: External File List Slot</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name Offset in Local Heap</td>
|
|
<td>The byte offset within the local name heap for the name of the file. File names are stored as a URL
|
|
which has a protocol name, a host name, a port number, and a file name:
|
|
<code><em>protocol</em>:<em>port</em>//<em>host</em>/<em>file</em></code>. If the protocol is omitted
|
|
then “file:” is assumed. If the port number is omitted then a default port for that protocol
|
|
is used. If both the protocol and the port number are omitted then the colon can also be omitted. If the double
|
|
slash and host name are omitted then “localhost” is assumed. The file name is the only mandatory part,
|
|
and if the leading slash is missing then it is relative to the application’s current working directory
|
|
(the use of relative names is not recommended).</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Offset in External Data File</td>
|
|
<td>This is the byte offset to the start of the data in the specified file. For files that contain data for
|
|
a single dataset this will usually be zero.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Data Size in External File</td>
|
|
<td>This is the total number of bytes reserved in the specified file for raw data storage. For a file that
|
|
contains exactly one complete dataset which is not extendable, the size will usually be the exact size of
|
|
the dataset. However, by making the size larger one allows HDF5 to extend the dataset. The size can be set
|
|
to a value larger than the entire file since HDF5 will read zeros past the end of the file without failing.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_layout IV.A.3.i. The Data Layout Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Data Storage - Layout</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0008</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Required for datasets; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The Data Layout message describes how the elements of a multi-dimensional array
|
|
are stored in the HDF5 file. Four types of data layout are supported:
|
|
<ol>
|
|
<li>Contiguous: The array is stored in one contiguous area of the file. This layout requires that the
|
|
size of the array be constant: data manipulations such as chunking, compression, checksums or encryption
|
|
are not permitted. The message stores the total storage size of the array. The offset of an element from
|
|
the beginning of the storage area is computed as in a C array.</li>
|
|
<li>Chunked: The array domain is regularly decomposed into chunks, and each chunk is allocated and stored
|
|
separately. This layout supports arbitrary element traversals, compression, encryption, and checksums
|
|
(these features are described in other messages). The message stores the size of a chunk instead of the
|
|
size of the entire array; the size of the entire array can be calculated by traversing the B-tree that
|
|
stores the chunk addresses.</li>
|
|
<li>Compact: The array is stored in one contiguous block, as part of this object header messagei.</li>
|
|
<li>Virtual: This is only supported for version 4 of the Data Layout message. The message stores information
|
|
that is used to locate the global heap collection containing the Virtual Dataset (VDS) mapping information.
|
|
The mapping associates the VDS to the source dataset elements that are stored across a collection
|
|
of HDF5 files.</li>
|
|
</ol></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Data Layout Message (Versions 1 and 2)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Dimensionality</td>
|
|
<td>Layout Class</td>
|
|
<td>Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data Address<sup>O</sup> <em>(optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#0 Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#1 Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#(d-1) Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dataset Element Size <em>(stored as Dimension \#d)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Compact Data Size <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Compact Data...<em>(variable size, optional)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Data Layout Message (Versions 1 and 2)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of the data layout message and
|
|
is described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Used by version 1.4 and before of the library to encode layout information. Data space is
|
|
always allocated when the data set is created.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Used by version 1.6.x of the library to encode layout information. Data space is allocated
|
|
only when it is necessary.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimensionality</td>
|
|
<td>\anchor FMT4V12Dimensionality An array has a fixed dimensionality.
|
|
Let <b>d</b> be the number of dimensions in the dataset’s dataspace (its rank).
|
|
This field stores the number of entries in the Dimension Extent list below.
|
|
<br />
|
|
For compact and contiguous storage, that list has exactly <b>d</b> entries.
|
|
For chunked storage, the list has <b>d + 1</b> entries: the first <b>d</b> (Dimension \#0 Extent
|
|
through Dimension \#(d-1) Extent) are the chunk’s dimension extents, and one additional
|
|
trailing entry — not a real dimension — holds the Dataset Element Size (see below).
|
|
For example, a 1-dimensional chunked dataset stores a Dimensionality value of 2.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Layout Class</td>
|
|
<td>The layout class specifies the type of storage for the data and how the other fields of the layout
|
|
message are to be interpreted.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Compact Storage</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Contiguous Storage</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Chunked Storage</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Data Address</td>
|
|
<td>For contiguous storage, this is the address of the raw data in the file. For chunked storage this is
|
|
the address of the @ref subsubsec_fmt4_infra_btrees_v1 that is used to look up the addresses of the
|
|
chunks. This field is not present for compact storage. If the version for this message is greater than 1,
|
|
the address may have the @ref FMT4UndefinedAddress "undefined address" value, to indicate that storage
|
|
has not yet been allocated for this array.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dimension \#i Extent (i = 0 to (d-1))</td>
|
|
<td>\anchor FMT4V12DimInfoN For contiguous and compact storage the dimensions define the entire extent
|
|
of the array, in units of array elements (not bytes).
|
|
Dimension \#0 is the slowest-changing axis and Dimension \#(d-1) is the fastest-changing axis.
|
|
<br />For chunked storage, the dimensions define the extent of a single chunk, in units of array
|
|
elements (not bytes).
|
|
Dimension \#0 is the slowest-changing axis and Dimension \#(d-1) is the fastest-changing axis.
|
|
The following entry, Dimension \#d, is not a chunk dimension; it holds the Dataset Element Size
|
|
(see below).</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataset Element Size</td>
|
|
<td>\anchor FMT4V12DsetElmSz The size of a dataset element, in bytes.
|
|
This field is only present for chunked data storage.
|
|
Although it is not a dimension extent, it is encoded as the trailing entry, Dimension \#d, in the
|
|
Dimension Extent list above, which is why the Dimensionality field stores d + 1 rather than d
|
|
for chunked storage.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Compact Data Size</td>
|
|
<td>This field is only present for compact data storage. It contains the size of the raw data for the
|
|
dataset array, in bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Compact Data</td>
|
|
<td>This field is only present for compact data storage. It contains the raw data for the dataset
|
|
array.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
Version 3 of this message re-structured the format into specific properties that are required for each layout class.
|
|
<table>
|
|
<caption><strong>Layout: Data Layout Message (Version 3)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Layout Class</td>
|
|
<td colspan="2" bgcolor=#DDDDDD><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Properties <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Data Layout Message (Version 3)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of layout message and is
|
|
described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Used by the version 1.6.3 and later of the library to store properties for each layout class.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Layout Class</td>
|
|
<td>The layout class specifies how the other fields of the layout message are to be interpreted.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Compact Storage</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Contiguous Storage</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Chunked Storage</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Properties</td>
|
|
<td>This variable-sized field encodes information specific to each layout class and is described below. If
|
|
there is no property information specified for a layout class, the size of this field is zero bytes.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4CompactStorage <h4>Class-specific information for compact layout (Class 0):</h4><br />
|
|
(Note: The dimensionality information is in the Dataspace message)
|
|
<table>
|
|
<caption><strong>Layout: Compact Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Size</td>
|
|
<td colspan="2" bgcolor=#DDDDDD><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Raw Data...<em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Compact Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Size</td>
|
|
<td>This field contains the size of the raw data for the dataset array, in bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Raw Data</td>
|
|
<td>This field contains the raw data for the dataset array.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ContiguousStorage <h4>Class-specific information for contiguous storage (layout class 1):</h4><br />
|
|
(Note: The dimensionality information is in the Dataspace message)
|
|
|
|
<table>
|
|
<caption><strong>Layout: Contiguous Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Contiguous Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This is the address of the raw data in the file. The address may have the
|
|
@ref FMT4UndefinedAddress "undefined address" value, to indicate that storage has not
|
|
yet been allocated for this array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size</td>
|
|
<td>This field contains the size allocated to store the raw data, in bytes.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
Class-specific information for chunked storage (layout class 2):
|
|
<table>
|
|
<caption><strong>Layout: Chunked Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td colspan="3" bgcolor=#DDDDDD><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#0 Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#1 Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">...</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dimension \#(d-1) Extent</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Dataset Element Size <em>(stored as Dimension \#d)</em></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li In the above table, <b>d</b> is the number of dimensions in the dataset’s dataspace;
|
|
see @ref FMT4V12Dimensionality "Dimensionality" in versions 1 and 2 Data Layout message, in particular the chunked storage portion.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Chunked Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td>\anchor FMT4Dimensionality See @ref FMT4V12Dimensionality "Dimensionality" description in versions 1 and 2 Data Layout message, in particular the chunked storage portion.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This is the address of the @ref subsubsec_fmt4_infra_btrees_v1 that is used to look up the
|
|
addresses of the chunks that actually store portions of the array data. The address may have the
|
|
@ref FMT4UndefinedAddress "undefined address" value, to indicate
|
|
that storage has not yet been allocated for this array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension \#i Extent (i = 0 to (d-1))</td>
|
|
<td>\anchor FMT4DimInfoN See @ref FMT4V12DimInfoN "Dimension \#i Extent" description in versions 1 and 2 Data Layout message, in particular the chunked storage portion.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataset Element Size</td>
|
|
<td>\anchor FMT4DsetElmSz See @ref FMT4V12DsetElmSz "Dataset Element Size" description in versions 1 and 2 Data Layout message.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4DataLayoutV4 <h4>Version 4 of this message is similar to version 3 but has additional
|
|
information for the virtual layout class as well as indexing information for the chunked layout class.</h4>
|
|
<h4>Version 5 of this message is identical to version 4, except for the encoding of filtered chunk records for Fixed Array, Extensible Array, and v2 B-Tree indices. The Library also requires layout version 5 for datasets with chunk size greater than or equal to 4 GiB.</h4>
|
|
<table>
|
|
<caption><strong>Layout: Data Layout Message (Versions 4 and 5)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Layout Class</td>
|
|
<td colspan="2" bgcolor=#DDDDDD><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Properties <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Data Layout Message (Versions 4 and 5)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of the data layout message and
|
|
is described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>Used by version 1.10.0 and later of the library to store properties for each layout class and indexing information for the chunked layout.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>Used by version 2.0.0 and later of the library and is identical to version 4, except the encoding of filtered chunk records for Fixed Array, Extensible Array, and v2 B-Tree indices has been changed to allow for filters that expand chunks up to 2<sup>64</sup>-1 bytes. The Library will also not allow creation of datasets with chunk size greater than 2<sup>32</sup>-1 bytes (4 GiB - 1) with layout version 4 or below (including unfiltered chunks), because versions of the library that don't understand layout version 5 cannot process these large chunks. However, such large chunks are not precluded by the version 4 layout encoding per se.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Layout Class</td>
|
|
<td>The layout class specifies the type of storage for the data and how the other fields
|
|
of the layout message are to be interpreted.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Compact Storage</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Contiguous Storage</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Chunked Storage</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Virtual Storage</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Properties</td>
|
|
<td>This variable-sized field encodes information specific to each layout class as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Layout Class</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Compact Storage</td>
|
|
<td>See @ref FMT4CompactStorage "Compact Storage Property Description" for the version 3
|
|
Data Layout message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Contiguous Storage</td>
|
|
<td>See @ref FMT4ContiguousStorage "Contiguous Storage Property Description" for the version 3
|
|
Data Layout message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Chunked Storage</td>
|
|
<td>See @ref FMT4ChunkedStorage "Chunked Storage Property Description" below.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left">Virtual Storage</td>
|
|
<td>See @ref FMT4VirtualStorage "Virtual Storage Property Description" below.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4ChunkedStorage <h4>Class-specific information for chunked storage (layout class 2):</h4>
|
|
<table>
|
|
<caption><strong>Layout: Chunked Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>Dimensionality</td>
|
|
<td>Dimension Extent Encoded Length</td>
|
|
<td bgcolor=#DDDDDD><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#0 Extent <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#1 Extent <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />...<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dimension \#(d-1) Extent <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dataset Element Size <em>(stored as Dimension \#d)</em> <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Chunk Indexing Type</td>
|
|
<td colspan="3" bgcolor=#DDDDDD><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Indexing Type Information <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li In the above table, <b>d</b> is the number of dimensions in the dataset’s dataspace;
|
|
see @ref FMT4V12Dimensionality "Dimensionality" in versions 1 and 2 Data Layout message, in particular the chunked storage portion.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Chunked Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This is the chunked layout feature flag:
|
|
<table>
|
|
<tr>
|
|
<th width="55%" align="left">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="left"><code>DONT_FILTER_PARTIAL_BOUND_CHUNKS (bit 0)</code></td>
|
|
<td>Do not apply filter to a partial edge chunk.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left"><code>SINGLE_INDEX_WITH_FILTER (bit 1)</code></td>
|
|
<td>A filtered chunk for <i>Single Chunk</i> indexing.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimensionality</td>
|
|
<td>See @ref FMT4V12Dimensionality "Dimensionality" description in versions 1 and 2 Data Layout message, in particular the chunked storage portion.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension Extent Encoded Length</td>
|
|
<td>This is the size in bytes used to encode each <em>Dimension Extent</em> value.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dimension \#i Extent (i = 0 to (d-1))</td>
|
|
<td>See @ref FMT4V12DimInfoN "Dimension \#i Extent" description in versions 1 and 2 Data Layout message, in particular the chunked storage portion.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataset Element Size</td>
|
|
<td>See @ref FMT4V12DsetElmSz "Dataset Element Size" description in versions 1 and 2 Data Layout message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Chunk Indexing Type</td>
|
|
<td>There are five indexing types used to look up addresses of the chunks. For more information on each
|
|
type, see @ref sec_fmt4_appendixc<br />
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>@ref subsec_fmt4_appendixc_chunk indexing type.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>@ref subsec_fmt4_appendixc_implicit indexing type.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>@ref subsec_fmt4_appendixc_fixedarr indexing type.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>@ref subsec_fmt4_appendixc_extarr indexing type.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>@ref subsec_fmt4_appendixc_appv2btree indexing type.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Indexing Type Information</td>
|
|
<td>This variable-sized field encodes information specific to an indexing type. More information on
|
|
what is encoded with each type can be found below this table.
|
|
<ul>
|
|
<li>See <em>@ref FMT4IndexInfoSingle "Single Chunk"</em> below.</li>
|
|
<li>See <em>@ref FMT4IndexInfoImplicit "Implicit"</em> below.</li>
|
|
<li>See <em>@ref FMT4IndexInfoFixed "Fixed Array"</em> below.</li>
|
|
<li>See <em>@ref FMT4IndexInfoExtensible "Extensible Array"</em> below.</li>
|
|
<li>See <em>@ref FMT4IndexInfoV2Btrees "Version 2 B-tree"</em> below.</li>
|
|
</ul></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This is the address specific to an indexing type. The address may be undefined if the chunk or
|
|
index storage is not allocated yet.<br />
|
|
<table>
|
|
<tr>
|
|
<th width="40%" align="left">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="left"><i>Single Chunk index</i></td>
|
|
<td align="left">Address of the single chunk.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left"><i>Implicit index</i></td>
|
|
<td align="left">Address of the array of dataset chunks.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left"><i>Fixed Array index</i></td>
|
|
<td align="left">Address of the index.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left"><i>Extensible Array index</i></td>
|
|
<td align="left">Address of the index.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="left"><i>Version 2 B-tree index</i></td>
|
|
<td align="left">Address of the index.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<ol>
|
|
<li>\anchor FMT4IndexInfoSingle <h4>Index-specific information for <i>Single Chunk</i>:</h4>
|
|
The following information exists only when the chunk is filtered. In other words, when
|
|
<code>SINGLE_INDEX_WITH_FILTER</code> (bit 1) is enabled in the field <em>flags</em>.
|
|
<table>
|
|
<caption><strong>Layout: Single Chunk Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of filtered chunk<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filters for chunk</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Single Chunk Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of filtered chunk</td>
|
|
<td>This field is the size of a filtered chunk.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filters for chunk</td>
|
|
<td>This field contains filters for the chunk.</td>
|
|
</tr>
|
|
</table>
|
|
</li>
|
|
|
|
<li>\anchor FMT4IndexInfoImplicit <h4>Index-specific information for <i>Implicit</i>:</h4>
|
|
<table>
|
|
<caption><strong>Layout: Implicit Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><em>No specific indexing information</em></td>
|
|
</tr>
|
|
</table>
|
|
</li>
|
|
|
|
<li>\anchor FMT4IndexInfoFixed <h4>Index-specific information for <i>Fixed Array</i>:</h4>
|
|
<table>
|
|
<caption><strong>Layout: Fixed Array Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Page Bits</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
</table>
|
|
<table>
|
|
<caption><strong>Fields: Fixed Array Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Page Bits</td>
|
|
<td>This field contains the number of bits needed to store the maximum number of elements in a
|
|
data block page.</td>
|
|
</tr>
|
|
</table>
|
|
</li>
|
|
|
|
<li>\anchor FMT4IndexInfoExtensible <h4>Index-specific information for <i>Extensible Array</i>:</h4>
|
|
<table>
|
|
<caption><strong>Layout: Extensible Array Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Max Bits</td>
|
|
<td>Index Elements</td>
|
|
<td>Min Pointers</td>
|
|
<td>Min Elements</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Page Bits</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Extensible Array Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Max Bits</td>
|
|
<td>This field contains the number of bits needed to store the maximum number of elements
|
|
in the array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Index Elements</td>
|
|
<td>This field contains the number of elements to store in the index block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Min Pointers</td>
|
|
<td>This field contains the minimum number of data block pointers for a superblock.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Min Elements</td>
|
|
<td>This field contains the minimum number of elements per data block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Page Bits</td>
|
|
<td>This field contains the number of bits needed to store the maximum number of elements in
|
|
a data block page.</td>
|
|
</tr>
|
|
</table>
|
|
</li>
|
|
|
|
<li>\anchor FMT4IndexInfoV2Btrees <h4>Index-specific information for <i>Version 2 B-tree</i>:</h4>
|
|
<table>
|
|
<caption><strong>Layout: Version 2 B-tree Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Node Size</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Split Percent</td>
|
|
<td>Merge Percent</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
</table>
|
|
<table>
|
|
<caption><strong>Fields: Version 2 B-tree Indexing Information</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Node Size</td>
|
|
<td>This field is the size in bytes of a B-tree node.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Split Percent</td>
|
|
<td>This field is the percentage full of a B-tree node at which to split the node.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Merge Percent</td>
|
|
<td>This field is the percentage full of a B-tree node at which to merge the node.</td>
|
|
</tr>
|
|
</table>
|
|
</li>
|
|
</ol>
|
|
|
|
\anchor FMT4VirtualStorage <h4>Class-specific information for virtual storage (layout class 3):</h4>
|
|
<table>
|
|
<caption><strong>Layout: Virtual Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Index</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Virtual Storage Property Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>This is the address of the global heap collection where the VDS mapping entries are stored.
|
|
See @ref subsec_fmt4_infra_globalheapvds</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Index</td>
|
|
<td>This is the index of the data object within the global heap collection.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_bogus IV.A.3.j. The Bogus Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Bogus</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b>0x0009</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> 4 bytes</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> For testing only; should never be stored in a valid file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message is used for testing the HDF5 Library’s response to an “unknown”
|
|
message type and should never be encountered in a valid HDF5 file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Bogus Message</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Bogus Value</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Bogus Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Bogus Value</td>
|
|
<td>This value should always be: <code>0xdeadbeef</code>.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_groupinfo IV.A.3.k. The Group Info Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Group Info</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x000A</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Required for “new style” groups; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message stores information for the constants defining a “new style” group’s
|
|
behavior. Constant information will be stored in this message and variable information will be stored
|
|
in the @ref subsubsec_fmt4_dataobject_hdr_msg_linkinfo message.<br />
|
|
Note: the “estimated entry” information below is used when determining the size of the
|
|
object header for the group when it is created.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Group Info Message</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td colspan="2">Link Phase Change: Maximum Compact Value <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Link Phase Change: Minimum Dense Value <em>(optional)</em></td>
|
|
<td colspan="2">Estimated Number of Entries <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Estimated Link Name Length of Entries <em>(optional)</em></td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Group Info Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This is the group information flag with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, link phase change values are stored.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, the estimated entry information is non-default
|
|
and is stored.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-7</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link Phase Change: Maximum Compact Value</td>
|
|
<td>This is the maximum number of links to store “compactly” (in the group’s object header).
|
|
If this field is not present, the default value of <code>8</code> will be used for the maximum compact
|
|
value.<br />
|
|
This field is present if bit 0 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Link Phase Change: Minimum Dense Value</td>
|
|
<td>This is the minimum number of links to store “densely” (in the group’s fractal
|
|
heap). The fractal heap’s address is located in the @ref subsubsec_fmt4_dataobject_hdr_msg_linkinfo
|
|
message. If this field is not present, the default value of <code>6</code> will be used for the minimum
|
|
dense value.<br />
|
|
This field is present if bit 0 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Estimated Number of Entries</td>
|
|
<td>This is the estimated number of entries in groups. If this field is not present, the default value of
|
|
<code>4</code> will be used for the estimated number of group entries.<br />
|
|
This field is present if bit 1 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Estimated Link Name Length of Entries</td>
|
|
<td>This is the estimated length of entry name. If this field is not present, the default value of
|
|
<code>8</code> will be used for the estimated link name length of group entries.<br />
|
|
This field is present if bit 1 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_filter IV.A.3.l. The Data Storage - Filter Pipeline Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Data Storage - Filter Pipeline</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x000B</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message describes the filter pipeline which should be applied to the data stream by providing filter
|
|
identification numbers, flags, a name, and client data.<br />
|
|
This message may be present in the object headers of both dataset and group objects. For datasets, it
|
|
specifies the filters to apply to raw data. For groups, it specifies the filters to apply to the
|
|
group’s fractal heap. Currently, only datasets using chunked data storage use the filter pipeline
|
|
on their raw data.<br />
|
|
This message may be shared between multiple objects in the file; see @ref subsec_fmt4_dataobject_hdr_msg
|
|
for the shared message encoding.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Filter Pipeline Message - Version 1</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td>Number of Filters</td>
|
|
<td colspan="2">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Reserved (zero)</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Filter Description List <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Filter Pipeline Message - Version 1</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This table describes version 1.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Number of Filters</td>
|
|
<td>The total number of filters described in this message. The maximum possible number of filters in a
|
|
message is 32.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Filter Description List</td>
|
|
<td>A description of each filter. A filter description appears in the next table.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Filter Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Filter Identification</td>
|
|
<td colspan="2">Name Length</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Flags</td>
|
|
<td colspan="2">Number of Values for Client Data</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Name <em>(variable size, optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Client Data <em>(variable size, optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Padding <em>(variable size, optional)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Filter Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Filter Identification Value</td>
|
|
<td>This value, often referred to as a filter identifier, is designed to be a unique identifier for
|
|
the filter. Values from zero through 32,767 are reserved for filters supported by The HDF Group
|
|
in the HDF5 library and for filters requested and supported by third parties. Filters supported
|
|
by The HDF Group are documented immediately below. Information on 3rd-party filters can be found
|
|
at The HDF Group’s <a href="https://\PLURL/docs/RegisteredFilterPlugins.md">
|
|
Registered Filters</a> page.<br />
|
|
<a href="#FMT4Footnote1Change"><sup><small>1</small></sup></a><br /> To request a filter identifier,
|
|
please contact The HDF Group’s Help Desk at <a href="mailto:help@hdfgroup.org">HDF Help Desk</a>.
|
|
You will be asked to provide the following information:
|
|
<ol>
|
|
<li>Contact information for the developer requesting the new identifier</li>
|
|
<li>A short description of the new filter</li>
|
|
<li>Links to any relevant information, including licensing information</li>
|
|
</ol><br />
|
|
Values from 32768 to 65535 are reserved for non-distributed uses (for example, internal company usage)
|
|
or for application usage when testing a feature. The HDF Group does not track or document the use of
|
|
the filters with identifiers from this range.<br />
|
|
The filters currently in library version 1.8.0 are listed below:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Identification</th>
|
|
<th width="15%" align="left">Name</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>N/A</td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>deflate</td>
|
|
<td>GZIP deflate compression</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>shuffle</td>
|
|
<td>Data element shuffling</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>fletcher32</td>
|
|
<td>Fletcher32 checksum</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>szip</td>
|
|
<td>SZIP compression</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>nbit</td>
|
|
<td>N-bit packing</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6</code></td>
|
|
<td>scaleoffset</td>
|
|
<td>Scale and offset encoded values</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name Length</td>
|
|
<td>Each filter has an optional null-terminated ASCII name and this field holds the length of the name
|
|
including the null termination padded with nulls to be a multiple of eight. If the filter has no name
|
|
then a value of zero is stored in this field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Flags</td>
|
|
<td>The flags indicate certain properties for a filter. The bit values defined so far are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set then the filter is an optional filter. During output, if an optional filter fails it will be
|
|
silently skipped in the pipeline.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1-15</code></td>
|
|
<td>Reserved (zero)</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Number of Client Data Values</td>
|
|
<td>Each filter can store integer values to control how the filter operates. The number of entries
|
|
in the <em>Client Data</em> array is stored in this field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name</td>
|
|
<td>If the <em>Name Length</em> field is non-zero then it will contain the size of this field, padded
|
|
to a multiple of eight. This field contains a null-terminated, ASCII character string to serve as
|
|
a comment/name for the filter.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Client Data</td>
|
|
<td>This is an array of four-byte integers which will be passed to the filter function. The <em>Client Data
|
|
Number of Values</em> determines the number of elements in the array.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Padding</td>
|
|
<td>Four bytes of zeroes are added to the message at this point if the Client Data Number of Values field
|
|
contains an odd number.</td>
|
|
</tr>
|
|
</table>
|
|
\anchor FMT4Footnote1Change <sup>1</sup> If you are reading an earlier version of this document, this
|
|
link may have changed. If the link does not work, use the latest version of this document on
|
|
<a href="https://github.com/HDFGroup/hdf5">The HDF Group</a>’s github website,
|
|
\ref SPEC; the link there will always be correct.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Filter Pipeline Message - Version 2</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td>Number of Filters</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Filter Description List <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
u
|
|
<table>
|
|
<caption align="top"><strong>Fields: Filter Pipeline Message - Version 2</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This table describes version 2.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Number of Filters</td>
|
|
<td>The total number of filters described in this message. The maximum possible number of filters in a
|
|
message is 32.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Filter Description List</td>
|
|
<td>A description of each filter. A filter description appears in the next table.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Filter Description</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Filter Identification</td>
|
|
<td colspan="2">Name Length <em>(optional)</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Flags</td>
|
|
<td colspan="2">Number Client Data Values</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Name <em>(variable size, optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Client Data <em>(variable size, optional)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Filter Description</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Filter Identification Value</td>
|
|
<td>This value, often referred to as a filter identifier, is designed to be a unique identifier for
|
|
the filter. Values from zero through 32,767 are reserved for filters supported by The HDF Group
|
|
in the HDF5 library and for filters requested and supported by third parties. Filters supported
|
|
by The HDF Group are documented immediately below. Information on 3rd-party filters can be found
|
|
at The HDF Group’s <a href="https://\PLURL/docs/RegisteredFilterPlugins.md">
|
|
Registered Filters</a> page.<br />
|
|
<a href="#FMT4Footnote1Change"><sup><small>1</small></sup></a><br /> To request a filter identifier,
|
|
please contact The HDF Group’s Help Desk at <a href="mailto:help@hdfgroup.org">HDF Help Desk</a>.
|
|
You will be asked to provide the following information:
|
|
<ol>
|
|
<li>Contact information for the developer requesting the new identifier</li>
|
|
<li>A short description of the new filter</li>
|
|
<li>Links to any relevant information, including licensing information</li>
|
|
</ol><br />
|
|
Values from 32768 to 65535 are reserved for non-distributed uses (for example, internal company usage)
|
|
or for application usage when testing a feature. The HDF Group does not track or document the use of
|
|
the filters with identifiers from this range.<br />
|
|
The filters currently in library version 1.8.0 are listed below:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Identification</th>
|
|
<th width="15%" align="left">Name</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>N/A</td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>deflate</td>
|
|
<td>GZIP deflate compression</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>shuffle</td>
|
|
<td>Data element shuffling</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>fletcher32</td>
|
|
<td>Fletcher32 checksum</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>szip</td>
|
|
<td>SZIP compression</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code></td>
|
|
<td>nbit</td>
|
|
<td>N-bit packing</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>6</code></td>
|
|
<td>scaleoffset</td>
|
|
<td>Scale and offset encoded values</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name Length</td>
|
|
<td>Each filter has an optional null-terminated ASCII name and this field holds the length of the name
|
|
including the null termination. If the filter has no name
|
|
then a value of zero is stored in this field.<br />
|
|
Filters with IDs less than 256 (in other words, filters that are defined in this format documentation)
|
|
do not store the <em>Name Length</em> or <em>Name</em> fields.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Flags</td>
|
|
<td>The flags indicate certain properties for a filter. The bit values defined so far are:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set then the filter is an optional filter. During output, if an optional filter fails it will be
|
|
silently skipped in the pipeline.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1-15</code></td>
|
|
<td>Reserved (zero)</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Number of Client Data Values</td>
|
|
<td>Each filter can store integer values to control how the filter operates. The number of entries
|
|
in the <em>Client Data</em> array is stored in this field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name</td>
|
|
<td>If the <em>Name Length</em> field is non-zero then it will contain the size of this field, not padded
|
|
to a multiple of eight. This field contains a null-terminated, ASCII character string to serve as
|
|
a comment/name for the filter.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Client Data</td>
|
|
<td>This is an array of four-byte integers which will be passed to the filter function. The <em>Client Data
|
|
Number of Values</em> determines the number of elements in the array.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_attribute IV.A.3.m. The Attribute Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Attribute</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x000C</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The <em>Attribute</em> message is used to store objects in the HDF5 file which are used as attributes,
|
|
or “metadata” about the current object. An attribute is a small dataset; it has a name,
|
|
a datatype, a dataspace, and raw data. Since attributes are stored in the object header, they should
|
|
be relatively small (in other words, less than 64KB). They can be associated with any type of object
|
|
which has an object header (groups, datasets, or committed (named) datatypes).<br />
|
|
In 1.8.x versions of the library, attributes can be larger than 64KB. See the
|
|
“ @ref subsec_attribute_special ” section of the Attributes chapter in
|
|
the <cite>@ref UG</cite> for more information.<br />
|
|
Note: Attributes on an object must have unique names: the HDF5 Library currently enforces this by
|
|
causing the creation of an attribute with a duplicate name to fail. Attributes on different
|
|
objects may have the same name, however.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Attribute Message (Version 1)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td>Reserved (zero)</td>
|
|
<td colspan="2">Name Size</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Datatype Size</td>
|
|
<td colspan="2">Dataspace Size</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Name <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Datatype <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dataspace <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Data <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Attribute Message (Version 1)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of the attribute message and is
|
|
described here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Used by the library before version 1.6 to encode attribute message. This version does not
|
|
support shared datatypes.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name Size</td>
|
|
<td>The length of the attribute name in bytes including the null terminator. Note that the
|
|
<em>Name</em> field below may contain additional padding not represented by this field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Datatype Size</td>
|
|
<td>The length of the datatype description in the <em>Datatype</em> field below. Note that the
|
|
<em>Datatype</em> field may contain additional padding not represented by this field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Dataspace Size</td>
|
|
<td>The length of the dataspace description in the <em>Dataspace</em> field below. Note that the
|
|
<em>Dataspace</em> field may contain additional padding not represented by this field.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name</td>
|
|
<td>The null-terminated attribute name. This field is padded with additional null characters to make it a
|
|
multiple of eight bytes.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Type</td>
|
|
<td>The datatype description follows the same format as described for the datatype object header message.
|
|
This field is padded with additional zero bytes to make it a multiple of eight bytes.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Space</td>
|
|
<td>The dataspace description follows the same format as described for the dataspace object header message.
|
|
This field is padded with additional zero bytes to make it a multiple of eight bytes.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Data</td>
|
|
<td>The raw data for the attribute. The size is determined from the datatype and dataspace descriptions.
|
|
This field is <em>not</em> padded with additional bytes.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Attribute Message (Version 2)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td colspan="2">Name Size</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Datatype Size</td>
|
|
<td colspan="2">Dataspace Size</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Name <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Datatype <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dataspace <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Attribute Message (Version 2)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of the attribute message and is
|
|
described here:
|
|
<table width="90%">
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Used by the library of version 1.6.x and after to encode attribute messages. This version
|
|
supports shared datatypes. The fields of name, datatype, and dataspace are not padded with
|
|
additional bytes of zero.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2Flags Flags</td>
|
|
<td>This bit field contains extra information about interpreting the attribute message:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Bit</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, datatype is shared.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, dataspace is shared.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2NameSize Name Size</td>
|
|
<td>The length of the attribute name in bytes including the null terminator.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2DatatypeSize Datatype Size</td>
|
|
<td>The length of the datatype description in the <em>Datatype</em> field below.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2DataspaceSize Dataspace Size</td>
|
|
<td>The length of the dataspace description in the <em>Dataspace</em> field below.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2Name Name</td>
|
|
<td>The null-terminated attribute name. This field is <em>not</em> padded with additional bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2Datatype Datatype</td>
|
|
<td>The datatype description follows the same format as described for the datatype object
|
|
header message.<br />
|
|
If the <em>Flag</em> field indicates this attribute’s datatype is shared, this field will
|
|
contain a @ref subsec_fmt4_dataobject_hdr_msg "shared message encoding" instead of
|
|
the datatype encoding.<br />
|
|
This field is <em>not</em> padded with additional bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2Dataspace Dataspace</td>
|
|
<td>The dataspace description follows the same format as described for the dataspace object
|
|
header message.<br />
|
|
If the <em>Flag</em> field indicates this attribute’s dataspace is shared, this field will
|
|
contain a @ref subsec_fmt4_dataobject_hdr_msg "shared message encoding" instead of
|
|
the dataspace encoding.<br />
|
|
This field is <em>not</em> padded with additional bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\anchor FMT4AttrV2Data Data</td>
|
|
<td>The raw data for the attribute. The size is determined from the datatype and dataspace
|
|
descriptions.<br />
|
|
This field is <em>not</em> padded with additional zero bytes.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Attribute Message (Version 3)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td colspan="2">Name Size</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Datatype Size</td>
|
|
<td colspan="2">Dataspace Size</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Name Character Set Encoding</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Name <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Datatype <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dataspace <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Attribute Message (Version 3)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number information is used for changes in the format of the attribute message and is
|
|
described here:
|
|
<table width="90%">
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Used by the library of version 1.8.x and after to encode attribute messages. This version
|
|
supports attributes with non-ASCII names.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2Flags "version 2" of the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Name Size</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2NameSize "version 2" of the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Datatype Size</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2DatatypeSize "version 2" of the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataspace Size</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2DataspaceSize "version 2" of the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Name Character Set Encoding</td>
|
|
<td>The character set encoding for the attribute’s name:<br />
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>ASCII character set encoding</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>UTF-8 character set encoding</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Name</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2Name "version 2" of the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Datatype</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2Datatype "version 2" of the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataspace</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2Dataspace "version 2" of the attribute message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data</td>
|
|
<td>This field is the same as described for @ref FMT4AttrV2Data "version 2" of the attribute message.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_comment IV.A.3.n. The Object Comment Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Object Comment</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x000D</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The object comment is designed to be a short description of an object. An object comment is a sequence
|
|
of non-zero (<code>\0</code>) ASCII characters with no other formatting included by the library.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Object Comment Message</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Comment <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Object Comment Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Name</td>
|
|
<td>A null terminated ASCII character string.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_omodified IV.A.3.o. The Object Modification Time (Old) Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Object Modification Time (Old)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x000E</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The object modification date and time is a timestamp which indicates (using ISO-8601 date and
|
|
time format) the last modification of an object. The time is updated when any object header
|
|
message changes according to the system clock where the change was posted. All fields of this
|
|
message should be interpreted as coordinated universal time (UTC).<br />
|
|
This modification time message is deprecated in favor of the “new”
|
|
@ref subsubsec_fmt4_dataobject_hdr_msg_mod message and is no longer written to the file in
|
|
versions of the HDF5 Library after the 1.6.0 version.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Modification Time Message (Old)</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Year</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Month</td>
|
|
<td colspan="2">Day of Month</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Hour</td>
|
|
<td colspan="2">Minute</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Second</td>
|
|
<td colspan="2">Reserved</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Modification Time Message (Old)</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Year</td>
|
|
<td>The four-digit year as an ASCII string. For example, <code>1998</code>.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Month</td>
|
|
<td>The month number as a two digit ASCII string where January is <code>01</code> and December is
|
|
<code>12</code>.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Day of Month</td>
|
|
<td>The day number within the month as a two digit ASCII string. The first day of the month is
|
|
<code>01</code>.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Hour</td>
|
|
<td>The hour of the day as a two digit ASCII string where midnight is <code>00</code> and 11:00pm
|
|
is <code>23</code>.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Minute</td>
|
|
<td>The minute of the hour as a two digit ASCII string where the first minute of the hour is
|
|
<code>00</code> and the last is <code>59</code>.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Second</td>
|
|
<td>The second of the minute as a two digit ASCII string where the first second of the minute is
|
|
<code>00</code> and the last is <code>59</code>.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Reserved</td>
|
|
<td>This field is reserved and should always be zero.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_shared IV.A.3.p. The Shared Message Table Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Shared Message Table</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x000F</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message is used to locate the @ref subsec_fmt4_infra_sohm "SOHM Master Table". Each
|
|
index in the table consists of information to find the shared messages from either the heap or
|
|
object header. This message is <em>only</em> found in the superblock extension.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption align="top"><strong>Layout: Shared Message Table Message</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Shared Object Header Message Table Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Number of Indices</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption align="top"><strong>Fields: Shared Message Table Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 0.</td>
|
|
</tr>
|
|
<tr valign=top>
|
|
<td>Shared Object Header Message Table Address</td>
|
|
<td>This field is the address of the master table for shared object header message indexes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Number of Indices</td>
|
|
<td>This field is the number of indices in the master table.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_continuation IV.A.3.q. The Object Header Continuation Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Object Header Continuation</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0010</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The object header continuation is the location in the file of a block containing more header messages
|
|
for the current data object. This can be used when header blocks become too large or are likely to
|
|
change over time.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Object Header Continuation Message</strong></caption>
|
|
<tr align="center">
|
|
<th width=25%>byte</th>
|
|
<th width=25%>byte</th>
|
|
<th width=25%>byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Offset<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Length<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Object Header Continuation Message</strong></caption>
|
|
<tr>
|
|
<th width=30%>Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Offset</td>
|
|
<td>This value is the address in the file where the header continuation block is located.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length</td>
|
|
<td>This value is the length in bytes of the header continuation block in the file.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
The format of the header continuation block that this message points to depends on the version of the
|
|
object header that the message is contained within.
|
|
|
|
Continuation blocks for version 1 object headers have no special formatting information; they are
|
|
merely a list of object header message info sequences (type, size, flags, reserved bytes and data for
|
|
each message sequence). See the description of @ref subsubsec_fmt4_dataobject_hdr_prefix_one.
|
|
|
|
Continuation blocks for version 2 object headers <em>do</em> have special formatting information as
|
|
described here (see also the description of @ref subsubsec_fmt4_dataobject_hdr_prefix_two):
|
|
|
|
<table>
|
|
<caption><strong>Layout: Version 2 Object Header Continuation Block</strong></caption>
|
|
<tr>
|
|
<th width=25%>byte</th>
|
|
<th width=25%>byte</th>
|
|
<th width=25%>byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message Type \#1</td>
|
|
<td colspan="2">Size of Header Message Data \#1</td>
|
|
<td>Header Message \#1 Flags</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Header Message \#1 Creation Order <em>(optional)</em></td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Message Data \#1<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message Type \#n</td>
|
|
<td colspan="2">Size of Header Message Data \#n</td>
|
|
<td>Header Message \#n Flags</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Header Message \#n Creation Order <em>(optional)</em></td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Message Data \#n<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Gap <em>(optional, variable size)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 Object Header Continuation Block</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>OCHK</code>” is used to indicate the
|
|
beginning of an object header continuation block. This gives file consistency checking
|
|
utilities a better chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Type</td>
|
|
<td>Same format as version 1 of the object header, described above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Header Message \#n Data</td>
|
|
<td>Same format as version 1 of the object header, described above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Flags</td>
|
|
<td>Same format as version 1 of the object header, described above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Creation Order</td>
|
|
<td>This field stores the order that a message of a given type was created in.<br />
|
|
This field is present if bit 2 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Message \#n Data</td>
|
|
<td>Same format as version 1 of the object header, described above.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Gap</td>
|
|
<td>A gap in this @ref FMT4OhdrChunkDef "continuation chunk" is inferred by the end of the messages for
|
|
the chunk before the beginning of this chunk’s Checksum field. Gaps are always smaller than the
|
|
size of an object header message prefix (message type + message size + message flags).<br />
|
|
Gaps are formed when a message (typically an attribute message) in an earlier chunk is deleted
|
|
and a message from a later chunk that does not quite fit into the free space is moved into the
|
|
earlier chunk.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>This is the checksum for this continuation chunk only (see
|
|
@ref FMT4OhdrChunkDef "object header chunks"), computed over every byte of this chunk from the
|
|
<code>OCHK</code> signature through the end of this chunk’s Gap field (if present), excluding
|
|
this Checksum field itself. Chunk \#0, in the object header prefix, and every other continuation
|
|
chunk each carry their own separate checksum.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_stmgroup IV.A.3.r. The Symbol Table Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Symbol Table Message</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0011</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Required for “old style” groups; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>Each “old style” group has a v1 B-tree and a local heap for storing symbol table entries,
|
|
which are located with this message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Symbol Table Message</strong></caption>
|
|
<tr align="center">
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />v1 B-tree Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Local Heap Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Symbol Table Message</strong></caption>
|
|
<tr>
|
|
<th width=30%>Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>v1 B-tree Address</td>
|
|
<td>This value is the address of the v1 B-tree containing the symbol table entries for the group.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Local Heap Address</td>
|
|
<td>This value is the address of the local heap containing the link names for the symbol table
|
|
entries for the group.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_mod IV.A.3.s. The Object Modification Time Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Object Modification Time</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0012</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>The object modification time is a timestamp which indicates the time of the last modification of
|
|
an object. The time is updated when any object header message changes according to the system clock
|
|
where the change was posted.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Modification Time Message</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3">Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Seconds After UNIX Epoch</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Modification Time Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number is used for changes in the format of Object Modification Time and is described
|
|
here:
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Never used.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Used by Version 1.6.1 and after of the library to encode time. In this version, the time is
|
|
the seconds after Epoch.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Seconds After UNIX Epoch</td>
|
|
<td>A 32-bit unsigned integer value that stores the number of seconds since 0 hours, 0 minutes,
|
|
0 seconds, January 1, 1970, Coordinated Universal Time.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_btreek IV.A.3.t. The B-tree ‘K’ Values Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> B-tree ‘K’ Values</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0013</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message retrieves non-default ‘K’ values for internal and leaf nodes of a group
|
|
or indexed storage v1 B-trees. This message is <em>only</em> found in the superblock extension.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: B-tree ‘K’ Values Message</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="2">Indexed Storage Internal Node K</td>
|
|
<td bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Group Internal Node K</td>
|
|
<td colspan="2">Group Leaf Node K</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: B-tree ‘K’ Values Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Indexed Storage Internal Node K</td>
|
|
<td>This is the node ‘K’ value for each internal node of an indexed storage v1 B-tree.
|
|
See the description of this field in version 0 and 1 of the superblock as well the section on
|
|
v1 B-trees.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Group Internal Node K</td>
|
|
<td>This is the node ‘K’ value for each internal node of a group v1 B-tree. See the
|
|
description of this field in version 0 and 1 of the superblock as well as the section
|
|
on v1 B-trees.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Group Leaf Node K</td>
|
|
<td>This is the node ‘K’ value for each leaf node of a group v1 B-tree. See the
|
|
description of this field in version 0 and 1 of the superblock as well as the section on v1
|
|
B-trees.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_drvinfo IV.A.3.u. The Driver Info Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Driver Info</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0014</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message contains information needed by the file driver to reopen a file. This message is
|
|
<em>only</em> found in the superblock extension: see the @ref subsec_fmt4_boot_supext section
|
|
for more information. For more information on the fields in the driver info message, see the
|
|
@ref subsec_fmt4_boot_driver section; those who use the multi and family file drivers will find
|
|
this section particularly helpful.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Driver Info Message</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Driver Identification</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Driver Information Size</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br /><br />Driver Information <em>(variable size)</em><br /><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Driver Info Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Driver Identification</td>
|
|
<td>This is an eight-byte ASCII string without null termination which identifies the driver.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Driver Information Size</td>
|
|
<td>The size in bytes of the <em>Driver Information</em> field of this message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Driver Information</td>
|
|
<td>Driver information is stored in a format defined by the file driver.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_attrinfo IV.A.3.v. The Attribute Info Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Attribute Info</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0015</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Varies</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message stores information about the attributes on an object, such as the maximum creation
|
|
index for the attributes created and the location of the attribute storage when the attributes
|
|
are stored “densely”.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Attribute Info Message</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Flags</td>
|
|
<td colspan="2">Maximum Creation Index <em>(optional)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Fractal Heap Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Attribute Name v2 B-tree Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Attribute Creation Order v2 B-tree Address<sup>O</sup> <em>(optional)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Attribute Info Message</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document
|
|
describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This is the attribute index information flag with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, creation order for attributes is tracked.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>If set, creation order for attributes is indexed.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2-7</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Maximum Creation Index</td>
|
|
<td>The is the maximum creation order index value for the attributes on the object.<br />
|
|
This field is present if bit 0 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Fractal Heap Address</td>
|
|
<td>This is the address of the fractal heap to store dense attributes. Each attribute stored in the
|
|
fractal heap is described by the @ref subsubsec_fmt4_dataobject_hdr_msg_attribute.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Attribute Name v2 B-tree Address</td>
|
|
<td>This is the address of the version 2 B-tree to index the names of densely stored attributes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Attribute Creation Order v2 B-tree Address</td>
|
|
<td>This is the address of the version 2 B-tree to index the creation order of densely stored
|
|
attributes.<br />
|
|
This field is present if bit 1 of <em>Flags</em> is set.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_refcount IV.A.3.w. The Object Reference Count Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Object Reference Count</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0016</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message stores the number of hard links (in groups or objects) pointing to an object: in
|
|
other words, its <em>reference count</em>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Object Reference Count</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Reference count</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Object Reference Count</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Reference Count</td>
|
|
<td>The unsigned 32-bit integer is the reference count for the object. This message is only present
|
|
in “version 2” (or later) object headers, and if not present in those object header versions,
|
|
the reference count for the object is assumed to be 1.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_fsinfo IV.A.3.x. The File Space Info Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> File Space Info</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0017</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message stores the file space management information that the library uses in handling file
|
|
space requests for the file. Version 0 of the message is used for release 1.10.0 only. Version 1
|
|
of the message is used for release 1.10.1+. There is no File Space Info message before release
|
|
1.10 as the library does not track file space across multiple file opens.<br />
|
|
Note that version 0 is deprecated starting in release 1.10.1. That means when the 1.10.1+ library
|
|
opens an HDF5 file with a version 0 message, the library will decode and map the message to
|
|
version 1. On file close, it will encode the message as a version 1 message.<br />
|
|
The library uses the following three mechanisms to manage file space in an HDF5 file:
|
|
<ul>
|
|
<li> Free-space managers<br />
|
|
They track free-space sections of various sizes in the file that are not currently
|
|
allocated. Each free-space manager corresponds to a file space type. There are two main
|
|
groups of file space types: metadata and raw data. Metadata is further divided into five
|
|
types: superblock, B-tree, global heap, local heap, and object header. See the description
|
|
of @ref subsec_fmt4_infra_freespaceindex as well the description of file space allocation
|
|
types in @ref sec_fmt4_appendixb.</li>
|
|
<li> Aggregators<br />
|
|
The library manages two aggregators, one for metadata and one for raw data. Aggregator is
|
|
a contiguous block of free-space in the file. The size of each aggregator is tunable via
|
|
public routines #H5Pset_meta_block_size and #H5Pset_small_data_block_size respectively.</li>
|
|
<li> Virtual file drivers<br />
|
|
The library's virtual file driver interface dispatches requests for additional space to the
|
|
allocation routine of the file driver associated with the file. For example, if the sec2
|
|
file driver is being used, its allocation routine will increase the size of the file to
|
|
service the requests.</li>
|
|
</ul>
|
|
For release 1.10.0, the library derives the following four file space strategies based on the mechanisms:
|
|
<ul>
|
|
<li>#H5F_FILE_SPACE_ALL
|
|
<ul>
|
|
<li>Mechanisms used: free-space managers, aggregators, and virtual file drivers</li>
|
|
<li>Does not persist free-space across file opens</li>
|
|
<li>This strategy is the library default</li>
|
|
</ul>
|
|
</li>
|
|
<li>#H5F_FILE_SPACE_ALL_PERSIST</li>
|
|
<ul>
|
|
<li>Mechanisms used: free-space managers, aggregators, and virtual file drivers</li>
|
|
<li>Persist free-space across file opens</li>
|
|
</ul>
|
|
<li>#H5F_FILE_SPACE_AGGR_VFD</li>
|
|
<ul>
|
|
<li>Mechanisms used: aggregators and virtual file drivers</li>
|
|
<li>Does not persist free-space across file opens</li>
|
|
</ul>
|
|
<li>#H5F_FILE_SPACE_VFD</li>
|
|
<ul>
|
|
<li>Mechanisms used: virtual file drivers</li>
|
|
<li>Does not persist free-space across file opens</li>
|
|
</ul>
|
|
</ul>
|
|
For release 1.10.1+, the free-space manager mechanism is modified to handle paged aggregation
|
|
which aggregates small metadata and raw data allocations into constant-sized well-aligned pages
|
|
to allow efficient I/O accesses. With the support of this feature, the library derives the
|
|
following four file space strategies:
|
|
<ul>
|
|
<li>#H5F_FSPACE_STRATEGY_FSM_AGGR </li>
|
|
<ul>
|
|
<li>Mechanisms used: free-space managers, aggregators, and virtual file drivers</li>
|
|
<li>This strategy is the library default</li>
|
|
</ul>
|
|
<li>#H5F_FSPACE_STRATEGY_PAGE</li>
|
|
<ul>
|
|
<li>Mechanisms used: free-space managers with embedded paged aggregation and virtual file drivers</li>
|
|
</ul>
|
|
<li>#H5F_FSPACE_STRATEGY_AGGR</li>
|
|
<ul>
|
|
<li>Mechanisms used: aggregators and virtual file drivers</li>
|
|
</ul>
|
|
<li>#H5F_FSPACE_STRATEGY_NONE</li>
|
|
<ul>
|
|
<li>Mechanisms used: virtual file drivers</li>
|
|
</ul>
|
|
</ul>
|
|
The default is not persisting free-space across file opens for the above four strategies. User can use
|
|
the public routine #H5Pset_file_space_strategy to request persisting free-space.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: File Space Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Strategy</td>
|
|
<td colspan="2">Threshold<sup>L</sup></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Free-space manager address<sup>O</sup> for #H5FD_MEM_SUPER<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Free-space manager address<sup>O</sup> for #H5FD_MEM_BTREE<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Free-space manager address<sup>O</sup> for #H5FD_MEM_DRAW<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Free-space manager address<sup>O</sup> for #H5FD_MEM_GHEAP<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Free-space manager address<sup>O</sup> for #H5FD_MEM_LHEAP<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Free-space manager address<sup>O</sup> for #H5FD_MEM_OHDR<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: File Space Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This is the version 0 of this message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Strategy</td>
|
|
<td>This is the file space strategy used to manage file space. There are four types:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>#H5F_FILE_SPACE_ALL_PERSIST</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>#H5F_FILE_SPACE_ALL</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>#H5F_FILE_SPACE_AGGR_VFD</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>#H5F_FILE_SPACE_VFD</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Threshold</td>
|
|
<td>This is the smallest free-space section size that the free-space manager will track.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Free-space manager addresses</td>
|
|
<td>These are the six free-space manager addresses for the six file space allocation types:
|
|
<ul>
|
|
<li>#H5FD_MEM_SUPER</li>
|
|
<li>#H5FD_MEM_BTREE</li>
|
|
<li>#H5FD_MEM_DRAW</li>
|
|
<li>#H5FD_MEM_GHEAP</li>
|
|
<li>#H5FD_MEM_LHEAP</li>
|
|
<li>#H5FD_MEM_OHDR</li>
|
|
</ul>
|
|
Note that these six fields exist only if the value for the field “<em>Strategy</em>”
|
|
is #H5F_FILE_SPACE_ALL_PERSIST.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: File Space Info - Version 1</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Strategy</td>
|
|
<td>Persisting free-space</td>
|
|
<td bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Free-space Section Threshold<sup>L</sup></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">File Space Page Size</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Page-end Metadata threshold</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">EOA<sup>O</sup></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of small-sized free-space manager for #H5FD_MEM_SUPER<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of small-sized free-space manager for #H5FD_MEM_BTREE<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of small-sized free-space manager for #H5FD_MEM_DRAW<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of small-sized free-space manager for #H5FD_MEM_GHEAP<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of small-sized free-space manager for #H5FD_MEM_LHEAP<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of small-sized free-space manager for #H5FD_MEM_OHDR<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of large-sized free-space manager for #H5FD_MEM_SUPER<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of large-sized free-space manager for #H5FD_MEM_BTREE<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of large-sized free-space manager for #H5FD_MEM_DRAW<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of large-sized free-space manager for #H5FD_MEM_GHEAP<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of large-sized free-space manager for #H5FD_MEM_LHEAP<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup> of large-sized free-space manager for #H5FD_MEM_OHDR<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: File Space Info - Version 1</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This is the version 1 of this message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Strategy</td>
|
|
<td>This is the file space strategy used to manage file space. There are four types:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>#H5F_FSPACE_STRATEGY_FSM_AGGR</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>#H5F_FSPACE_STRATEGY_PAGE</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>#H5F_FSPACE_STRATEGY_AGGR</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>#H5F_FSPACE_STRATEGY_NONE</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Persisting free-space</td>
|
|
<td>True or false in persisting free-space.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Free-space Section Threshold</td>
|
|
<td>This is the smallest free-space section size that the free-space manager will track.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>File space page size</td>
|
|
<td>This is the file space page size, which is used when the paged aggregation feature is enabled.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Page-end metadata threshold</td>
|
|
<td>This is the smallest free-space section size at the end of a page that the free-space manager will
|
|
track. This is used when the paged aggregation feature is enabled.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>EOA</td>
|
|
<td>The EOA before the allocation of free-space manager header and section info for the self-referential
|
|
free-space managers when persisting free-space.<br />
|
|
Note that self-referential free-space managers are managers that involve file space allocation for
|
|
the managers' free-space header and section info.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Addresses of small-sized free-space managers</td>
|
|
<td>These are the addresses of the six small-sized free-space manager addresses for the six file space
|
|
allocation types:
|
|
<ul>
|
|
<li>#H5FD_MEM_SUPER</li>
|
|
<li>#H5FD_MEM_BTREE</li>
|
|
<li>#H5FD_MEM_DRAW</li>
|
|
<li>#H5FD_MEM_GHEAP</li>
|
|
<li>#H5FD_MEM_LHEAP</li>
|
|
<li>#H5FD_MEM_OHDR</li>
|
|
</ul>
|
|
Note that these six fields exist only if the value for the field
|
|
“<em>Persisting free-space</em>” is true.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Addresses of large-sized free-space managers</td>
|
|
<td>These are the addresses of the six large-sized free-space manager addresses for the six file space
|
|
allocation types:
|
|
<ul>
|
|
<li>#H5FD_MEM_SUPER</li>
|
|
<li>#H5FD_MEM_BTREE</li>
|
|
<li>#H5FD_MEM_DRAW</li>
|
|
<li>#H5FD_MEM_GHEAP</li>
|
|
<li>#H5FD_MEM_LHEAP</li>
|
|
<li>#H5FD_MEM_OHDR</li>
|
|
</ul>
|
|
Note that these six fields exist only if the value for the field
|
|
“<em>Persisting free-space</em>” is true.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsubsection subsubsec_fmt4_dataobject_hdr_msg_mdci IV.A.3.y. The Metadata Cache Image Message
|
|
<table>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Name:</b> Metadata Cache Image</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Header Message Type:</b> 0x0018</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Length:</b> Fixed</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Status:</b> Optional; may not be repeated.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><b>Description:</b></td>
|
|
<td>This message records the location and size of a metadata cache image block written to the file, when
|
|
generation of a metadata cache image is enabled for the file (see #H5Pset_mdc_image_config). The
|
|
metadata cache image block is a serialized copy of the contents of the metadata cache at file close,
|
|
which allows the library to reconstruct the cache directly from this block the next time the file is
|
|
opened, instead of reconstructing it by parsing the file's metadata. This can significantly reduce the
|
|
time needed to open a file containing many pieces of cached metadata.<br />
|
|
The message is written to the superblock extension only when the file is closed with a metadata
|
|
cache image pending, and is removed from the superblock extension once the library has read the
|
|
referenced block back in at the next file open when the file is opened for read/write access, at
|
|
which point the file space it occupied is reclaimed. If the file is opened read-only, the message
|
|
and image block are left unchanged. See @ref subsec_fmt4_infra_mdci for the format of the metadata
|
|
cache image block itself.</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2"><b>Format of Data:</b> See the tables below.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Metadata Cache Image</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Metadata Cache Image Block Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Metadata Cache Image Block Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Metadata Cache Image</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for this message. This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Metadata Cache Image Block Address</td>
|
|
<td>This is the address in the file of the metadata cache image block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Metadata Cache Image Block Size</td>
|
|
<td>This is the size, in bytes, of the metadata cache image block.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_dataobject_storage IV.B. Disk Format: Level 2B - Data Object Data Storage
|
|
The data for an object is stored separately from the header information in the file and may not actually
|
|
be located in the HDF5 file itself if the header indicates that the data is stored externally. The
|
|
information for each record in the object is stored according to the dimensionality of the object
|
|
(indicated in the dataspace header message). Multi-dimensional array data is stored in C order; in other
|
|
words, the “last” dimension changes fastest.
|
|
|
|
Data whose elements are composed of atomic datatypes are stored in IEEE format, unless
|
|
they are specifically defined as being stored in a different machine format with the architecture-type
|
|
information from the datatype header message. This means that each architecture will need to
|
|
[potentially] byte-swap data values into the internal representation for that particular machine.
|
|
|
|
Data with a variable-length datatype is stored in the global heap of the HDF5 file. Each element in
|
|
the data object storage is represented as a 4-byte unsigned integer sequence length followed by a
|
|
@ref FMT4GlobalHeapID "Global Heap ID" that identifies the heap object containing the actual
|
|
variable-length data. The sequence length indicates the number of base-type elements in the
|
|
variable-length sequence and is always stored as a 4-byte unsigned integer, regardless of the
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
Data whose elements are composed of reference datatypes are stored in several different ways depending
|
|
on the particular reference type involved. Object pointers are just stored as the offset of the
|
|
object header being pointed to with the size of the pointer being the same number of bytes as offsets
|
|
in the file.
|
|
|
|
Dataset region references are stored as a heap-ID which points to the following information within the
|
|
file-heap: an offset of the object pointed to, number-type information (same format as header message),
|
|
dimensionality information (same format as header message), sub-set start and end information (in other
|
|
words, a coordinate location for each), and field start and end names (in other words, a [pointer to the]
|
|
string indicating the first field included and a [pointer to the] string name for the last field).
|
|
|
|
Data of a compound datatype is stored as a contiguous stream of the items in the structure, with each
|
|
item formatted according to its datatype.<br />
|
|
Description of datatypes for variable-length, references and compound classes can be found in
|
|
@ref subsubsec_fmt4_dataobject_hdr_msg_dtmessage.<br />
|
|
Information about global heap and heap ID can be found in @ref subsec_fmt4_infra_globalheap..<br />
|
|
For reference datatype, see also the encoding description for @ref subsec_fmt4_appendixd_encoderv and
|
|
@ref subsec_fmt4_appendixd_encodedp in Appendix D.
|
|
|
|
|
|
\section sec_fmt4_appendixa V. Appendix A: Definitions
|
|
Definitions of various terms used in this document are included in this section.
|
|
<table>
|
|
<tr>
|
|
<th width="20%">Term</th>
|
|
<th>Definition</th>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Undefined Address</strong></td>
|
|
<td>\anchor FMT4UndefinedAddress The "undefined address" for a file is a file address with all bits
|
|
set: in other words, <code>0xffff...ff</code>.</td>
|
|
</tr>
|
|
<tr>
|
|
<td><strong>Unlimited Extent</strong></td>
|
|
<td>\anchor FMT4UnlimitedDim The "unlimited extent" for a dimension extent is a value with all bits set:
|
|
in other words, <code>0xffff...ff</code>.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\section sec_fmt4_appendixb VI. Appendix B: File Space Allocation Types
|
|
There are six basic types of file memory allocation as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="30%">Basic Allocation Type</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_SUPER</td>
|
|
<td>File space allocated for <em>Superblock.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_BTREE</td>
|
|
<td>File space allocated for <em>B-tree.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_DRAW</td>
|
|
<td>File space allocated for raw data.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_GHEAP</td>
|
|
<td>File space allocated for <em>Global Heap.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_LHEAP</td>
|
|
<td>File space allocated for <em>Local Heap.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_OHDR</td>
|
|
<td>File space allocated for <em>Object Header.</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
There are other file space allocation types that are mapped to the above six basic types
|
|
because they are similar in nature. The mapping and the corresponding description are
|
|
listed in the following two tables:
|
|
<table>
|
|
<tr>
|
|
<th width="30%">Basic Allocation Type</th>
|
|
<th>Mapping of Allocation Types to Basic Allocation Types</th>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_SUPER</td>
|
|
<td><em>none</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_BTREE</td>
|
|
<td>#H5FD_MEM_SOHM_INDEX</td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_DRAW</td>
|
|
<td>#H5FD_MEM_FHEAP_HUGE_OBJ</td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_GHEAP</td>
|
|
<td><em>none</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_LHEAP</td>
|
|
<td>#H5FD_MEM_FHEAP_DBLOCK, #H5FD_MEM_FSPACE_SINFO</td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_OHDR</td>
|
|
<td>#H5FD_MEM_FHEAP_HDR, #H5FD_MEM_FHEAP_IBLOCK, #H5FD_MEM_FSPACE_HDR, #H5FD_MEM_SOHM_TABLE</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<tr>
|
|
<th width="30%">Allocation Type</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_FHEAP_HDR</td>
|
|
<td>File space allocated for <em>Fractal Heap Header.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_FHEAP_DBLOCK</td>
|
|
<td>File space allocated for <em>Fractal Heap Direct Blocks.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_FHEAP_IBLOCK</td>
|
|
<td>File space allocated for <em>Fractal Heap Indirect Blocks.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_FHEAP_HUGE_OBJ</td>
|
|
<td>File space allocated for huge objects in the fractal heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_FSPACE_HDR</td>
|
|
<td>File space allocated for <em>Free-space Manager Header.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_FSPACE_SINFO</td>
|
|
<td>File space allocated for <em>Free-space Section List</em> of the free-space manager.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_SOHM_TABLE</td>
|
|
<td>File space allocated for <em>Shared Object Header Message Table.</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td>#H5FD_MEM_SOHM_INDEX</td>
|
|
<td>File space allocated for <em>Shared Message Record List.</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
\section sec_fmt4_appendixc VII. Appendix C: Types of Indexes for Dataset Chunks
|
|
For an HDF5 file without the latest format enabled, the library uses the
|
|
@ref subsubsec_fmt4_infra_btrees_v1 to index dataset chunks.<br />
|
|
For an HDF5 file with the latest format enabled, the library uses one of the following five
|
|
indexing types depending on a chunked dataset’s dimension specification and the way it
|
|
is extended.
|
|
|
|
\subsection subsec_fmt4_appendixc_chunk VII.A. The Single Chunk Index
|
|
The <i>Single Chunk</i> index can be used when the dataset fulfills the following condition:
|
|
<ul>
|
|
<li>the current, maximum, and chunk dimension sizes are all the same</li>
|
|
</ul>
|
|
|
|
The dataset has only one chunk, and the address of the single chunk is stored in the version 4
|
|
<i>Data Layout</i> message. See the @ref subsec_fmt4_appendixc_chunk layout and field description tables.
|
|
|
|
\subsection subsec_fmt4_appendixc_implicit VII.B. The Implicit Index
|
|
The <i>Implicit</i> index can be used when the dataset fulfills the following conditions:
|
|
<ul>
|
|
<li>fixed maximum dimension sizes</li>
|
|
<li>no filter applied to the dataset</li>
|
|
<li>the timing for the space allocation of the dataset chunks is
|
|
<code>#H5D_ALLOC_TIME_EARLY</code></li>
|
|
</ul>
|
|
|
|
Since the dataset’s dimension sizes are known and storage space is to be allocated early, an
|
|
array of dataset chunks are allocated based on the maximum dimension sizes when the dataset is created.
|
|
The base address of the array is stored in the version 4 <i>Data Layout</i> message. See the
|
|
@ref subsec_fmt4_appendixc_chunk layout layout and field description tables.<br />
|
|
When accessing a dataset chunk with a specified offset, the address of the chunk in the array is computed
|
|
as below:
|
|
\code
|
|
base address + (size of a chunk in bytes * chunk index associated with the offset)
|
|
\endcode
|
|
|
|
\anchor FMT4ChunkIndex A chunk index starts at 0 and increases according to the fastest changing
|
|
dimension, then the next fastest, and so on. The chunk index for a dataset chunk offset is computed as below:
|
|
<ol>
|
|
<li>Calculate the scaled offset for each dimension in <code>scaled_offset</code>:<br />
|
|
<code>scaled_offset = chunk_offset/chunk_dims</code></li>
|
|
<li>Calculate the # of chunks for each dimension in <code>nchunks</code>:<br />
|
|
<code>nchunks = (curr_dims + chunk_dims - 1)/chunk_dims</code></li>
|
|
<li>Calculate the down chunks for each dimension in <code>down_chunks</code>:<br />
|
|
<code>
|
|
// d is the number of dimensions
|
|
for(i = (int)(d-1), acc = 1; i >= 0; i--) {
|
|
down_chunks[i] = acc;
|
|
acc *= nchunks[i];
|
|
}
|
|
</code></li>
|
|
<li>Calculate the chunk index in <code>chunk_index</code>:<br />
|
|
<code>
|
|
// d is the number of dimensions
|
|
for(u = 0, chunk_index = 0; u < d; u++)
|
|
chunk_index += down_chunks[u] * scaled_offset[u];
|
|
</code></li>
|
|
</ol>
|
|
|
|
For example, for a 2-dimensional dataset with <code>curr_dims[4,5]</code> and
|
|
<code>chunk_dims[3,2]</code>, there will be a total of 6 chunks, with 3 chunks in the fastest
|
|
changing dimension and 2 chunks in the slowest changing dimension. See the figure below.
|
|
The chunk index for the chunk offset <code>[3,4]</code> is computed as below:
|
|
<ol>
|
|
<li><code>scaled_offset[0] = 1, scaled_offset[1] = 2</code></li>
|
|
<li><code>nchunks[0] = 2, nchunks[1] = 3</code></li>
|
|
<li><code>down_chunks[0] = 3, down_chunks[1] = 1</code></li>
|
|
<li><code>chunk_index = 5</code></li>
|
|
</ol>
|
|
|
|
<table>
|
|
<tr>
|
|
<td><strong>Figure 3:</strong> Implicit index chunk diagram</td>
|
|
</tr>
|
|
<tr>
|
|
<td>\image html FileFormatSpecChunkDiagram.jpg</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\subsection subsec_fmt4_appendixc_fixedarr VII.C. The Fixed Array Index
|
|
The <i>Fixed Array</i> index can be used when the dataset fulfills the following condition:
|
|
<ul>
|
|
<li>fixed maximum dimension sizes</li>
|
|
</ul>
|
|
|
|
Since the maximum number of chunks is known, an array of in-file-on-disk addresses based on the
|
|
maximum number of chunks is allocated when data is written to the dataset. To access a dataset
|
|
chunk with a specified offset, the @ref FMT4ChunkIndex "chunk index" associated with the offset
|
|
is calculated. The index is mapped into the array to locate the disk address for the chunk.<br />
|
|
The Fixed Array (FA) index structure provides space and speed improvements in locating chunks over
|
|
index structures that handle more dynamic data accesses like a
|
|
@ref subsec_fmt4_appendixc_appv2btree index.The entry into the Fixed Array is the Fixed Array
|
|
header which contains metadata about the entries stored in the array. The header contains a
|
|
pointer to a data block which stores the array of entries that describe the dataset chunks.
|
|
For greater efficiency, the array will be divided into multiple pages if the number of entries
|
|
exceeds a threshold value. The space for the data block and possibly data block pages are allocated
|
|
as a single contiguous block of space.<br />
|
|
The content of the data block depends on whether paging is activated or not. When paging is not
|
|
used, elements that describe the chunks are stored in the data block. If paging is turned on,
|
|
the data block contains a bitmap indicating which pages are initialized. Then subsequent data
|
|
block pages will contain the entries that describe the chunks.<br />
|
|
An entry describes either a filtered or non-filtered dataset chunk. The formats for both element
|
|
types are described below.<br />
|
|
<table>
|
|
<caption><strong>Layout: Fixed Array Header</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Client ID</td>
|
|
<td>Entry Size</td>
|
|
<td>Page Bits</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Max Num Entries<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data Block Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fixed Array Header</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>FAHD</code>”
|
|
is used to indicate the beginning of a Fixed Array header.
|
|
This gives file consistency checking utilities a better
|
|
chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Client ID</td>
|
|
<td>The ID for identifying the client of the Fixed Array:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Non-filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2+</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Entry Size</td>
|
|
<td>The size in bytes of an entry in the Fixed Array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Page Bits</td>
|
|
<td>The number of bits needed to store the maximum number of entries in a
|
|
@ref FMT4FADataBlockPage "data block page"</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Max Num Entries</td>
|
|
<td>The maximum number of entries in the Fixed Array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data Block Address</td>
|
|
<td>The address of the data block in the Fixed Array.</td>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for the header.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Fixed Array Data Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Client ID</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Page Bitmap <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Elements <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fixed Array Data Block</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>FADB</code>” is used to indicate the
|
|
beginning of a Fixed Array data block. This gives file consistency checking utilities a
|
|
better chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Client ID</td>
|
|
<td>The ID for identifying the client of the Fixed Array:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th width="80%" align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Non-filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Address</td>
|
|
<td>The address of the Fixed Array header. Principally used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Page Bitmap</td>
|
|
<td>A bitmap indicating which data block pages are initialized.<br />
|
|
Exists only if the data block is paged.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Elements</td>
|
|
<td>Contains the elements stored in the data block and exists only if the data block is not paged.
|
|
There are two element types:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>@ref FMT4FaNonFilterChunk "Non-filtered dataset chunks"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>@ref FMT4FaFilterChunk "Filtered dataset chunks"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for the Fixed Array data block.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>\anchor FMT4FADataBlockPage Layout: Fixed Array Data Block Page</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Elements <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Fixed Array Data Block Page</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Elements</td>
|
|
<td>Contains the elements stored in the data block page.
|
|
There are two element types:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>@ref FMT4FaNonFilterChunk "Non-filtered dataset chunks"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>@ref FMT4FaFilterChunk "Filtered dataset chunks"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for a Fixed Array data block page.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4FaNonFilterChunk
|
|
<table>
|
|
<caption><strong> Layout: Data Block Element for Non-filtered Dataset Chunk</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Data Block Element for Non-filtered Dataset Chunk</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>The address of the dataset chunk in the file.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4FaFilterChunk
|
|
<table>
|
|
<caption><strong>Layout: Data Block Element for Filtered Dataset Chunk (Layout Version 4)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Chunk Size <em>(variable size; at most 8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li This encoding is used when this Fixed Array is used as an index for a dataset with layout message version 4.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Data Block Element for Filtered Dataset Chunk (Layout Version 5)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Chunk Size<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li This encoding is used when this Fixed Array is used as an index for a dataset with layout message version 5 or above.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Data Block Element for Filtered Dataset Chunk</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>The address of the dataset chunk in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Chunk Size</td>
|
|
<td>The size of the dataset chunk in bytes. The length of this field is determined depending on the version of the associated layout message as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Layout Message Version</th>
|
|
<th align="left">Chunk Size Encoding Length</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>One more than the number of bytes that would be needed to encode the size of an unfiltered chunk, but not more than 8.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code> or above</td>
|
|
<td>The size specified in the “@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filter Mask</td>
|
|
<td>Indicates the filter to skip for the dataset chunk. Each
|
|
filter has an index number in the pipeline; if that filter is
|
|
skipped, the bit corresponding to its index is set.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\section subsec_fmt4_appendixc_extarr VII.D. The Extensible Array Index
|
|
The <i>Extensible Array</i> index can be used when the dataset fulfills the following condition:
|
|
<ul>
|
|
<li>only one dimension of unlimited extent</li>
|
|
</ul>
|
|
|
|
The Extensible Array (EA) is a data structure that is used as a chunk index in datasets where the
|
|
dataspace has a single unlimited dimension. In other words, one dimension is set to
|
|
<code>H5S_UNLIMITED</code>, and the other dimensions are any number of fixed-size dimensions. The
|
|
idea behind the extensible array is that a particular data object can be located via a lightweight
|
|
indexing structure of fixed depth for a given address space. This indexing structure requires only
|
|
a few (2-3) file operations per element lookup and gives good cache performance. Unlike the B-tree
|
|
structure, the extensible array is optimized for appends. Where a B-tree would always add at the
|
|
rightmost node under these circumstances, either creating a deep tree (version 1) or requiring
|
|
expensive rebalances to correct (version 2), the extensible array has already mapped out a pre-balanced
|
|
internal structure. This optimized internal structure is instantiated as needed when chunk
|
|
records are inserted into the structure.<br />
|
|
An Extensible Array consists of a header, an index block, secondary blocks, data blocks, and
|
|
(optional) data block pages. The general scheme is that the index block is used to reference a
|
|
secondary block, which is, in turn, used to reference the data block page where the chunk information
|
|
is stored. The data blocks will be paged for efficiency when their size passes a threshold value.
|
|
These pages are laid out contiguously on the disk after the data block, are initialized as needed,
|
|
and are tracked via bitmaps stored in the secondary block. The number of secondary and data
|
|
blocks/pages in a chunk index varies as they are allocated as needed and the first few are
|
|
(conceptually) stored in parent elements as an optimization.
|
|
|
|
<table>
|
|
<caption><strong>
|
|
Layout: Extensible Array Header
|
|
</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Client ID</td>
|
|
<td>Element Size</td>
|
|
<td>Max Nelmts Bits</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Index Blk Elmts</td>
|
|
<td>Data Blk Min Elmts</td>
|
|
<td>Secondary Blk Min Data Ptrs</td>
|
|
<td>Max Data Blk Page Nelmts Bits</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Num Secondary Blks<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Secondary Blk Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Num Data Blks<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data Blk Size<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Max Index Set<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Num Elements<sup>L</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Index Block Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li Items marked with an ‘L’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Extensible Array Header</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>EAHD</code>” is used to indicate the beginning
|
|
of an Extensible Array header. This gives file consistency checking utilities a better chance
|
|
of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Client ID</td>
|
|
<td>The ID for identifying the client of the Fixed Array:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Non-filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Element Size</td>
|
|
<td>The size in bytes of an element in the Extensible Array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Max Nelmts Bits</td>
|
|
<td>The number of bits needed to store the maximum number of elements in the Extensible Array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Index Blk Elmts</td>
|
|
<td>The number of elements to store in the index block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data Blk Min Elmts</td>
|
|
<td>The minimum number of elements per data block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Secondary Blk Min Data Ptrs</td>
|
|
<td>The minimum number of data block pointers for a secondary block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Max Dblk Page Nelmts Bits</td>
|
|
<td>The number of bits needed to store the maximum number of elements in a data block page.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Num Secondary Blks</td>
|
|
<td>The number of secondary blocks created.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Secondary Blk Size</td>
|
|
<td>The size of the secondary blocks created.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Num Data Blks</td>
|
|
<td>The number of data blocks created.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data Blk Size</td>
|
|
<td>The size of the data blocks created.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Max Index Set</td>
|
|
<td>The maximum index set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Num Elmts</td>
|
|
<td>The number of elements realized.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Index Block Address</td>
|
|
<td>The address of the index block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for the header.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Extensible Array Index Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Client ID</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Elements <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data Block Addresses <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Secondary Block Addresses <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Extensible Array Index Block</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>EAIB</code>” is used to indicate the beginning
|
|
of an Extensible Array Index Block. This gives file consistency checking utilities a better
|
|
chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Client ID</td>
|
|
<td>The client ID for identifying the user of the Extensible Array:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Non-filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Address</td>
|
|
<td>The address of the Extensible Array header. Principally used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Elements</td>
|
|
<td>Contains the elements that are stored directly in the index block. An optimization to avoid unnecessary
|
|
secondary blocks.<br />There are two element types:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>@ref FMT4EaNonFilterChunk "Non-filtered dataset chunks"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>@ref FMT4EaFilterChunk "Filtered dataset chunks"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data Block Addresses</td>
|
|
<td>Contains the addresses of the data blocks that are stored directly in the Index Block. An
|
|
optimization to avoid unnecessary secondary blocks.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Secondary Block Addresses</td>
|
|
<td>Contains the addresses of the secondary blocks.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for the Extensible Array Index Block.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Extensible Array Secondary Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Client ID</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Block Offset <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Page Bitmap <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Data Block Addresses <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Extensible Array Secondary Block</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>EASB</code>” is used to indicate the beginning
|
|
of an Extensible Array Secondary Block. This gives file consistency checking utilities
|
|
a better chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Client ID</td>
|
|
<td>The ID for identifying the client of the Extensible Array:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Non-filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Address</td>
|
|
<td>The address of the Extensible Array header. Principally used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Block Offset</td>
|
|
<td>Stores the offset of the block in the array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Page Bitmap</td>
|
|
<td>A bitmap indicating which data block pages are initialized.<br />
|
|
Exists only if the data block is paged.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Data Block Addresses</td>
|
|
<td>Contains the addresses of the data blocks referenced by this secondary block.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for the Extensible Array Secondary Block.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Extensible Array Data Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Signature</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>Client ID</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Header Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Block Offset <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Elements <em>(variable size and optional)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Extensible Array Data Block</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Signature</td>
|
|
<td>The ASCII character string “<code>EADB</code>” is used to indicate the beginning
|
|
of an Extensible Array data block. This gives file consistency checking utilities a better
|
|
chance of reconstructing a damaged file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>This document describes version 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Client ID</td>
|
|
<td>The ID for identifying the client of the Extensible Array:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Non-filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Filtered dataset chunks</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2+</code></td>
|
|
<td>Reserved.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Header Address</td>
|
|
<td>The address of the Extensible Array header. Principally used for file integrity checking.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Block Offset</td>
|
|
<td>The offset of the block in the array.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Elements</td>
|
|
<td>Contains the elements stored in the data block and exists only if the data block is not paged.
|
|
<br />There are two element types:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>@ref FMT4EaNonFilterChunk "Non-filtered dataset chunks"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>@ref FMT4EaFilterChunk "Filtered dataset chunks"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for the Extensible Array data block.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Extensible Array Data Block Page</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Elements <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Checksum</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Fields: Extensible Array Data Block Page</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Elements</td>
|
|
<td>Contains the elements stored in the data block page.<br />
|
|
There are two element types:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">ID</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>@ref FMT4EaNonFilterChunk "Non-filtered dataset chunks"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>@ref FMT4EaFilterChunk "Filtered dataset chunks"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Checksum</td>
|
|
<td>The checksum for an Extensible Array data block page.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4EaNonFilterChunk
|
|
<table>
|
|
<caption><strong>Layout: Data Block Element for Non-filtered Dataset Chunk</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Data Block Element for Non-filtered Dataset Chunk</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>The address of the dataset chunk in the file.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4EaFilterChunk
|
|
<table>
|
|
<caption><strong>Layout: Data Block Element for Filtered Dataset Chunk (Layout Version 4)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Chunk Size<em> (variable size; at most 8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li This encoding is used when this B-Tree is used as an index for a dataset with layout message version 4.
|
|
|
|
|
|
<table>
|
|
<caption><strong>Layout: Data Block Element for Filtered Dataset Chunk (Layout Version 5)</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Chunk Size<em> (variable size; at most 8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Filter Mask</td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
\li This encoding is used when this B-Tree is used as an index for a dataset with layout message version 5 or above.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Data Block Element for Filtered Dataset Chunk</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Address</td>
|
|
<td>The address of the dataset chunk in the file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Chunk Size</td>
|
|
<td>The size of the dataset chunk in bytes. The length of this field is determined depending on the version of the associated layout message as follows:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Layout Message Version</th>
|
|
<th align="left">Chunk Size Encoding Length</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>One more than the number of bytes that would be needed to encode the size of an unfiltered chunk, but not more than 8.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>5</code> or above</td>
|
|
<td>The size specified in the “@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Filter Mask</td>
|
|
<td>Indicates the filter to skip for the dataset chunk.
|
|
Each filter has an index number in the pipeline; if that
|
|
filter is skipped, the bit corresponding to its index is set.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\section subsec_fmt4_appendixc_appv2btree VII.E. The Version 2 B-trees Index
|
|
The <i>Version 2 B-trees</i> index can be used when the dataset fulfills the following condition:
|
|
<ul>
|
|
<li>more than one dimension of unlimited extent</li>
|
|
</ul>
|
|
|
|
Version 2 B-trees can be used to index various objects in the library. See
|
|
@ref subsubsec_fmt4_infra_btrees_v2 for more information. The B-tree types
|
|
@ref FMT4V2BtType10 "10" and @ref FMT4V2BtType11 "11" record layouts are for
|
|
indexing dataset chunks.
|
|
|
|
\section sec_fmt4_appendixd VIII. Appendix D: Encoding for Dataspace, Datatype, and Reference
|
|
|
|
\section subsec_fmt4_appendixd_encode VIII.A. Dataspace Encoding
|
|
<i>#H5Sencode</i> is a public routine that encodes a dataspace description into a buffer while
|
|
<i>#H5Sdecode</i> is the corresponding routine that decodes the description encoded in the buffer.
|
|
See the reference manual description for these two public routines.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Dataspace Description for #H5Sencode/#H5Sdecode</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataspace ID</td>
|
|
<td>Encode Version</td>
|
|
<td>Size of Size</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Size of Extent<br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br /><br />Dataspace Message <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br /><br />Dataspace Selection <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong> Fields: Dataspace Description for #H5Sencode/#H5Sdecode</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataspace ID</td>
|
|
<td>The datspace message ID which is 1.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Encode Version</td>
|
|
<td>H5S_ENCODE_VERSION which is 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Size</td>
|
|
<td>The number of bytes used to store the size of an object.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Size of Extent</td>
|
|
<td>Size of the dataspace message.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataspace Message</td>
|
|
<td>The dataspace message information. See @ref subsubsec_fmt4_dataobject_hdr_msg_simple</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataspace Selection</td>
|
|
<td>The dataspace selection information. See @ref FMT4DataspaceSEL"Dataspace Selection".</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4DataspaceSEL
|
|
<table>
|
|
<caption><strong>Layout: Dataspace Selection</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Selection Type</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Selection Info (<em>variable size</em>)<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Fields: Dataspace Selection</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Selection Type</td>
|
|
<td>There are 4 types of selection:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>#H5S_SEL_NONE: Nothing selected</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>#H5S_SEL_POINTS: Sequence of points selected</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>#H5S_SEL_HYPERSLABS: Hyperslab selected</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>#H5S_SEL_ALL: Entire extent selected</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Selection Info</td>
|
|
<td>There are 4 types of selection info:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>Selection info for #H5S_SEL_NONE Layout and Fields @ref FMT4SelNONE "tables"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>Selection info for #H5S_SEL_POINTS Layout and Fields @ref FMT4SelPOINTS "tables"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>Selection info for #H5S_SEL_HYPERSLABS Layout and Fields @ref FMT4SelHYPER "tables"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>Selection for #H5S_SEL_ALL Layout and Fields @ref FMT4SelALL "tables"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelNONE
|
|
<table>
|
|
<caption><strong>Layout: Selection Info for #H5S_SEL_NONE</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Version</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Reserved <em>(zero, 8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Fields: Selection Info for #H5S_SEL_NONE</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for the #H5S_SEL_NONE Selection Info. The value is 1.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelPOINTS
|
|
<table>
|
|
<caption><strong>Layout: Selection Info for #H5S_SEL_POINTS</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Version</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br /><br />Points Selection Info <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Selection Info for #H5S_SEL_POINTS</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for the #H5S_SEL_POINTS Selection Info. The value is either 1 or 2.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Points Selection Info</td>
|
|
<td>Depending on <em>version</em>:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>See @ref FMT4SelPOINTSV1 "Version 1 Points Selection Info"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>See @ref FMT4SelPOINTSV2 "Version 2 Points Selection Info"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelPOINTSV1
|
|
<table>
|
|
<caption><strong>Layout: Version 1 Points Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Reserved <em>(zero)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Length</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Rank</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Num Points</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#1: coordinate \#1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#1: coordinate \#u</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#n: coordinate \#1</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#n: coordinate \#u</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 1 Points Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Length</td>
|
|
<td>The size in bytes from <em>Rank</em> to the end of the selection info.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Rank</td>
|
|
<td>The number of dimensions.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Num Points</td>
|
|
<td>The number of points in the selection.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Point \#n: coordinate \#u</td>
|
|
<td>The array of points in the selection. The points selected are \#1 to \#n where n is
|
|
<em>Num Points</em>. The list of coordinates for each point are \#1 to \#u where u is
|
|
<em>Rank</em>.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelPOINTSV2
|
|
<table>
|
|
<caption><strong>Layout: Version 2 Points Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="1">Encode Size</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em> </td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Rank</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Num Points(2, 4 or 8 bytes)<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#1: coordinate \#1(2, 4 or 8 bytes)<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#1: coordinate \#u(2, 4 or 8 bytes)<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#n: coordinate \#1 (2, 4 or 8 bytes)<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Point \#n: coordinate \#u(2, 4 or 8 bytes)<br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 Points Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Encode Size</td>
|
|
<td>The size for encoding the points selection info which can be 2, 4 or 8 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Rank</td>
|
|
<td>The number of dimensions.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Num Points</td>
|
|
<td>The number of points in the selection. The field <em>Encode Size</em> indicates the size
|
|
of this field</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Point \#n: coordinate \#u</td>
|
|
<td>The array of points in the selection. The points selected are \#1 to \#n where n is
|
|
<em>Num Points</em>. The list of coordinates for each point are \#1 to \#u where u is
|
|
<em>Rank</em>. The field <em>Encode Size</em> indicates the size of this field</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelHYPER
|
|
<table>
|
|
<caption><strong>Layout: Selection Info for #H5S_SEL_HYPERSLABS</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Version</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Hyperslab Selection Info (<em>variable size</em>)<br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Selection Info for #H5S_SEL_HYPERSLABS</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for the #H5S_SEL_HYPERSLABS selection info. The value is 1, 2 or 3.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Hyperslab Selection Info</td>
|
|
<td>Depending on <em>version</em>:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Version</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1</code></td>
|
|
<td>See @ref FMT4SelHYPERV1 "Version 1 Hyperslab Selection Info".</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>See @ref FMT4SelHYPERV2 "Version 2 Hyperslab Selection Info"</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>See @ref FMT4SelHYPERV3 "Version 3 Hyperslab Selection Info"</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelHYPERV1
|
|
<table>
|
|
<caption><strong>Layout: Version 1 Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Reserved</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Length</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Rank</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Num Blocks</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#0 for Block \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#(d-1) for Block \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#0 for Block \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#(d-1) for Block \#0</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#0 for Block \#(Num Blocks-1)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#(d-1) for Block \#(Num Blocks-1)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#0 for Block \#(Num Blocks-1)</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#(d-1) for Block \#(Num Blocks-1)</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 1 Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Length</td>
|
|
<td>The size in bytes from the field <em>Rank</em> to the end of the Selection Info.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Rank</td>
|
|
<td>The number of dimensions, <em>d</em>, in the dataspace.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Num Blocks</td>
|
|
<td>The number of blocks in the selection.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Starting Offset \#i for Block \#j (i = 0 to (d-1), j = 0 to (Num Blocks-1))</td>
|
|
<td>The offset of the starting element, along dimension \#i, of block \#j. Blocks are ordered
|
|
moving from the fastest changing dimension to the slowest changing dimension.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Ending Offset \#i for Block \#j (i = 0 to (d-1), j = 0 to (Num Blocks-1))</td>
|
|
<td>The offset of the ending element, along dimension \#i, of block \#j. Blocks are ordered
|
|
moving from the fastest changing dimension to the slowest changing dimension.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelHYPERV2
|
|
<table>
|
|
<caption><strong>Layout: Version 2 Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Length</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Rank</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Start \#0 <em>(8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Stride \#0 <em>(8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Count \#0 <em>(8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Block \#0 <em>(8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Start \#(d-1) <em>(8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Stride \#(d-1) <em>(8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Count \#(d-1) <em>(8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Block \#(d-1) <em>(8 bytes)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 2 Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This is a bit field with the following definition. Currently, this is always set to 0x1.
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, it is a regular hyperslab, otherwise, irregular.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length</td>
|
|
<td>The size in bytes from the field <em>Rank</em> to the end of the Selection Info.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Rank</td>
|
|
<td>The number of dimensions, <em>d</em>, in the dataspace.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Start \#i, Stride \#i, Count \#i, Block \#i (i = 0 to (d-1))</td>
|
|
<td>For each dimension \#i, these four values are stored together, in this order, before those of
|
|
dimension \#i+1:<br />
|
|
<em>Start \#i</em>: the offset of the starting element in the block, along dimension \#i.<br />
|
|
<em>Stride \#i</em>: the number of elements to move along dimension \#i.<br />
|
|
<em>Count \#i</em>: the number of blocks to select along dimension \#i.<br />
|
|
<em>Block \#i</em>: the extent (in elements) of each block along dimension \#i.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelHYPERV3
|
|
<table>
|
|
<caption><strong>Layout: Version 3 Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>Encode Size</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Rank</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Regular/Irregular Hyperslab Selection Info <em>(variable size)</em><br /><br/></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 3 Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This is a bit field with the following definition:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, it is a regular hyperslab, otherwise, irregular.</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Encode Size</td>
|
|
<td>The size for encoding hyperslab selection info, which can 2, 4 or 8 bytes.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Rank</td>
|
|
<td>The number of dimensions, <em>d</em>, in the dataspace.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Regular/Irregular Hyperslab Selection Info</td>
|
|
<td>This is the selection info for version 3 hyperslab which can be regular or irregular.
|
|
If bit 0 of the field <em>Flags</em> is set, see
|
|
@ref FMT4SelHYPERV3REG "Version 3 Regular Hyperslab Selection Info"
|
|
Otherwise, see @ref FMT4SelHYPERV3IRREG "Version 3 Irregular Hyperslab Selection Info"</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelHYPERV3REG
|
|
<table>
|
|
<caption><strong>Layout: Version 3 Regular Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Start \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Stride \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Count \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Block \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Start \#(d-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Stride \#(d-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Count \#(d-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Block \#(d-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 3 Regular Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Start \#i, Stride \#i, Count \#i, Block \#i (i = 0 to (d-1))</td>
|
|
<td>For each dimension \#i, these four values are stored together, in this order, before those of
|
|
dimension \#i+1:<br />
|
|
<em>Start \#i</em>: the offset of the starting element in the block, along dimension \#i.<br />
|
|
<em>Stride \#i</em>: the number of elements to move along dimension \#i.<br />
|
|
<em>Count \#i</em>: the number of blocks to select along dimension \#i.<br />
|
|
<em>Block \#i</em>: the extent (in elements) of each block along dimension \#i.<br />
|
|
The field <em>Encode Size</em> indicates the size of each of these values.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelHYPERV3IRREG
|
|
<table>
|
|
<caption><strong>Layout: Version 3 Irregular Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Num Blocks <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#0 for Block \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#(d-1) for Block \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#0 for Block \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#(d-1) for Block \#0 <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#0 for Block \#(Num Blocks-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Starting Offset \#(d-1) for Block \#(Num Blocks-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#0 for Block \#(Num Blocks-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">.<br />.<br />.<br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Ending Offset \#(d-1) for Block \#(Num Blocks-1) <em>(2, 4 or 8 bytes)</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Version 3 Irregular Hyperslab Selection Info</strong></caption>
|
|
<tr>
|
|
<td>Num Blocks</td>
|
|
<td>The number of blocks in the selection. The field <em>Encode Size</em> indicates the size of
|
|
this field</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Starting Offset \#i for Block \#j (i = 0 to (d-1), j = 0 to (Num Blocks-1))</td>
|
|
<td>The offset of the starting element, along dimension \#i, of block \#j. Blocks are ordered
|
|
moving from the fastest changing dimension to the slowest changing dimension. The field
|
|
<em>Encode Size</em> indicates the size of this field.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Ending Offset \#i for Block \#j (i = 0 to (d-1), j = 0 to (Num Blocks-1))</td>
|
|
<td>The offset of the ending element, along dimension \#i, of block \#j. Blocks are ordered
|
|
moving from the fastest changing dimension to the slowest changing dimension. The field
|
|
<em>Encode Size</em> indicates the size of this field.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\anchor FMT4SelALL
|
|
<table>
|
|
<caption><strong>Layout: Selection Info for #H5S_SEL_ALL</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4">Version</td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Reserved <em>(zero, 8 bytes)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Fields: Selection Info for #H5S_SEL_ALL</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Version</td>
|
|
<td>The version number for the #H5S_SEL_ALL Selection Info; the value is 1.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\section subsec_fmt4_appendixd_encodet VIII.B. Datatype Encoding
|
|
<i>#H5Tencode</i> is a public routine that encodes a datatype description into a buffer while
|
|
<i>#H5Tdecode</i> is the corresponding routine that decodes the description encoded in the buffer.
|
|
See the reference manual description for these two public routines.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Datatype Description for #H5Tencode/#H5Tdecode</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Datatype ID</td>
|
|
<td>Encode Version</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br /><br />Datatype Message <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Datatype Description for #H5Tencode/#H5Tdecode</strong></caption>
|
|
<tr>
|
|
<th width="40%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Datatype ID</td>
|
|
<td>The datatype message ID which is 3.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Encode Version</td>
|
|
<td>H5T_ENCODE_VERSION which is 0.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Datatype Message</td>
|
|
<td>The datatype message information. See @ref subsubsec_fmt4_dataobject_hdr_msg_dtmessage</td>
|
|
</tr>
|
|
</table>
|
|
|
|
\section subsec_fmt4_appendixd_encoderv VIII.C. Reference Encoding (Revised)
|
|
|
|
Reference information is stored as the dataset's raw data. The layout format is described below as
|
|
the Reference Header and the Reference Block.
|
|
|
|
<table>
|
|
<caption><strong>Layout: Reference Header</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Reference Type</td>
|
|
<td>Flags</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Reference Header</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Reference Type</td>
|
|
<td>There are 3 types of references:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Value</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>2</code></td>
|
|
<td>#H5R_OBJECT2: Object Reference</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>3</code></td>
|
|
<td>#H5R_DATASET_REGION2: Dataset Region Reference</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>4</code></td>
|
|
<td>#H5R_ATTR: Attribute Reference</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
<tr>
|
|
<td>Flags</td>
|
|
<td>This field describes the reference:
|
|
<table>
|
|
<tr>
|
|
<th width="20%" align="center">Bit</th>
|
|
<th align="left">Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>0</code></td>
|
|
<td>If set, the reference is to an external file.</td>
|
|
</tr>
|
|
<tr>
|
|
<td align="center"><code>1-7</code></td>
|
|
<td>Reserved</td>
|
|
</tr>
|
|
</table></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Reference Block</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td>Token Size</td>
|
|
<td colspan="3" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Token <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="2">Length of External File Name</td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />External File Name <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Size of Dataspace Selection</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Rank of Dataspace Selection</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Dataspace Selection Information <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="2">Length of Attribute Name </td>
|
|
<td colspan="2" bgcolor="#DDDDDD"><em>This space inserted only to align table nicely</em></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Attribute Name <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
|
|
<table>
|
|
<caption><strong>Fields: Reference Block</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Token size</td>
|
|
<td>This is the size of the token for the object.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Token</td>
|
|
<td>This is the token for the object.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length of External File Name</td>
|
|
<td>This is the length for the external file name.<br />
|
|
This field exists if bit 0 of <em>flags</em> is set.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>External File Name</td>
|
|
<td>This is the name of the external file being referenced.<br />
|
|
This field exists if bit 0 of <em>flags</em> is set.
|
|
</td>
|
|
<tr>
|
|
<td>Size of Dataspace Selection</td>
|
|
<td>This is the size of the Dataspace Selection Information.<br />
|
|
This field exists if the <em>Reference Type</em> is #H5R_DATASET_REGION2.
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Rank of Dataspace Selection</td>
|
|
<td>This is the number of dimensions in the dataspace.<br />
|
|
This field exists if the <em>Reference Type</em> is #H5R_DATASET_REGION2.
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>Dataspace Selection Information</td>
|
|
<td>See @ref FMT4DataspaceSEL "Dataspace Selection".<br />
|
|
This field exists if the <em>Reference Type</em> is #H5R_DATASET_REGION2.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Length of Attribute Name</td>
|
|
<td>This is the length of the attribute name.<br />
|
|
This field exists if the <em>Reference Type</em> is #H5R_ATTR.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Attribute Name</td>
|
|
<td>This is the name of the attribute being referenced.<br />
|
|
This field exists if the <em>Reference Type</em> is #H5R_ATTR.</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<br />
|
|
Depending on the <em>reference type</em> and the <em>flags</em> fields in the Reference Header,
|
|
the Reference Block is stored using one of two methods:
|
|
<ul>
|
|
<li>Direct method</li>
|
|
<br />The Reference Header and Reference Block are stored together directly as the dataset's raw data.
|
|
This method is used by:
|
|
<ul>
|
|
<li>Object Reference (H5R_OBJECT2) without reference to an external file</li>
|
|
</ul>
|
|
<li>Indirect method</li>
|
|
<br />The Reference Header, the size of the blob, and the Global Heap ID are stored as the dataset's raw data.
|
|
The blob size gives the size of the Reference Block stored in the global heap,
|
|
and the @ref FMT4GlobalHeapID "Global Heap ID" is used to locate it.
|
|
This method is used by:
|
|
<ul>
|
|
<li>Object Reference (#H5R_OBJECT2) (with reference to an external file)</li>
|
|
<li>Dataset Region Reference (#H5R_DATASET_REGION2) (with/without reference to an external file)</li>
|
|
<li>Attribute Reference (#H5R_ATTR) (with/without reference to an external file)</li>
|
|
</ul>
|
|
The layout for this method after the Reference Header is as follows:
|
|
|
|
<table>
|
|
<caption><strong>Layout: Indirect Method</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Blob Size</td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4"><br />Collection Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr align="center">
|
|
<td colspan="4">Object Index</td>
|
|
</tr>
|
|
|
|
</table>
|
|
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<br />
|
|
<table>
|
|
<caption><strong>Fields: Indirect Method</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Blob Size</td>
|
|
<td>This is the size of the Reference Block stored in the global heap.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Collection Address</td>
|
|
<td>See @ref FMT4GlobalHeapID "Global Heap ID".</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Object Index</td>
|
|
<td>See @ref FMT4GlobalHeapID "Global Heap ID".</td>
|
|
</tr>
|
|
</table>
|
|
|
|
</ul>
|
|
|
|
\section subsec_fmt4_appendixd_encodedp VIII.D. Reference Encoding (Backward Compatibility)
|
|
The two references described below are maintained to preserve compatibility with previous versions
|
|
of the library.<br />
|
|
For the following reference type, the reference encoding is stored as the dataset's raw data in the file:
|
|
<ul>
|
|
<li>Object Reference (#H5R_OBJECT1)</li>
|
|
</ul>
|
|
|
|
For the following reference type, the @ref FMT4GlobalHeapID "Global Heap ID" is stored as the dataset's
|
|
raw data in the file. The global heap ID is used to locate the reference encoding stored in the global heap:
|
|
<ul>
|
|
<li>Dataset Region Reference (#H5R_DATASET_REGION1)</li>
|
|
</ul>
|
|
|
|
<table>
|
|
<caption><strong>Layout: Reference for #H5R_OBJECT1</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Object Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Reference for #H5R_OBJECT1</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Object Address</td>
|
|
<td>Address of the object being referenced</td>
|
|
</tr>
|
|
</table>
|
|
<br />
|
|
<table>
|
|
<caption><strong>Layout: Reference for #H5R_DATASET_REGION1</strong></caption>
|
|
<tr>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th width="25%">byte</th>
|
|
<th>byte</th>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Object Address<sup>O</sup><br /><br /></td>
|
|
</tr>
|
|
<tr>
|
|
<td colspan="4"><br />Dataspace Selection Information <em>(variable size)</em><br /><br /></td>
|
|
</tr>
|
|
</table>
|
|
\li Items marked with an ‘O’ in the above table are of the size specified in
|
|
“@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock.
|
|
|
|
<table>
|
|
<caption><strong>Fields: Reference for #H5R_DATASET_REGION1</strong></caption>
|
|
<tr>
|
|
<th width="30%">Field Name</th>
|
|
<th>Description</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Object Address</td>
|
|
<td>This is the address of the object being referenced.</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Dataspace Selection Information</td>
|
|
<td>This is the dataspace selection for the object being referenced. See
|
|
@ref FMT4DataspaceSEL "Dataspace Selection".</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<hr>
|
|
Navigate back: \ref index "Main" / \ref SPEC
|
|
|
|
*/
|