From 972020e69fcc4c156cf64e63eef82e7c37c7be45 Mon Sep 17 00:00:00 2001 From: vchoi-hdfgroup <55293060+vchoi-hdfgroup@users.noreply.github.com> Date: Wed, 5 Aug 2026 17:41:26 -0500 Subject: [PATCH] Document the Metadata Cache Image block and header message (#6595) * Document the Metadata Cache Image block and header message Adds docs/doxygen/dox/H5.format.4.0.dox coverage for: - Level 1J: the Metadata Cache Image Block (subsec_fmt4_infra_mdci) - IV.A.3.y: the Metadata Cache Image header message (0x0018) - TOC/cross-reference entries and the superblock-extension list entry for both Also clarifies the 'Size of Header Message #n Data' field description for v1 and v2 object header messages to note it includes flags/reserved (v1) or flags/creation-order (v2) fields, not just type and length. * Address review feedback on MDCI message docs - Clarify that the Metadata Cache Image message and its image block are only removed/reclaimed on the next file open when the file is opened for read/write access; read-only opens leave them unchanged. - Fix a malformed table row (missing closing ) in the Status field of the IV.A.3.y message table. --- docs/doxygen/dox/H5.format.4.0.dox | 373 ++++++++++++++++++++++++++++- 1 file changed, 368 insertions(+), 5 deletions(-) diff --git a/docs/doxygen/dox/H5.format.4.0.dox b/docs/doxygen/dox/H5.format.4.0.dox index 2f51e43b5f9..ef0246744f9 100644 --- a/docs/doxygen/dox/H5.format.4.0.dox +++ b/docs/doxygen/dox/H5.format.4.0.dox @@ -35,6 +35,7 @@ Navigate back: \ref index "Main" / \ref SPEC
  • @ref subsec_fmt4_infra_fractalheap
  • @ref subsec_fmt4_infra_freespaceindex
  • @ref subsec_fmt4_infra_sohm
    • +
  • @ref subsec_fmt4_infra_mdci
  • @ref sec_fmt4_dataobject
      @@ -72,6 +73,7 @@ Navigate back: \ref index "Main" / \ref SPEC
    1. @ref subsubsec_fmt4_dataobject_hdr_msg_attrinfo
    2. @ref subsubsec_fmt4_dataobject_hdr_msg_refcount
    3. @ref subsubsec_fmt4_dataobject_hdr_msg_fsinfo
      4. +
    4. @ref subsubsec_fmt4_dataobject_hdr_msg_mdci
  • @ref subsec_fmt4_dataobject_storage
    • @@ -204,6 +206,7 @@ The following sections have been changed or added for the 1.10 release: \li @ref subsubsec_fmt4_dataobject_hdr_msg_layout section was changed. The name was changed, and @ref FMT4DataLayoutV4"version 4" of the data layout message was added for the virtual type. \li The @ref subsubsec_fmt4_dataobject_hdr_msg_fsinfo header message type was added. +\li The @ref subsubsec_fmt4_dataobject_hdr_msg_mdci header message type was added. \li @ref sec_fmt4_appendixc was added. Five indexing types were added. \section sec_fmt4_meta II. Disk Format: Level 0 - File Metadata @@ -792,6 +795,8 @@ later of the superblock is used. A superblock extension is an object header whic \li \ref subsubsec_fmt4_dataobject_hdr_msg_drvinfo containing information needed by the file driver in order to reopen a file. See also the \ref subsec_fmt4_boot_driver section above. \li \ref subsubsec_fmt4_dataobject_hdr_msg_fsinfo containing information about file space handling in the file. +\li \ref subsubsec_fmt4_dataobject_hdr_msg_mdci containing the location and size of a + @ref subsec_fmt4_infra_mdci "metadata cache image block", when one is present in the file. \section sec_fmt4_infra III. Disk Format: Level 1 - File Infrastructure @@ -4548,6 +4553,290 @@ The record for each shared message in an index is stored in one of the following +\subsection subsec_fmt4_infra_mdci III.J. Disk Format: Level 1J - Metadata Cache Image Block +The metadata cache image block is a serialized copy of the contents of the metadata cache at file +close, written when generation of a metadata cache image is enabled for the file (see +#H5Pset_mdc_image_config). It allows the library to reconstruct the cache directly from this block the +next time the file is opened, instead of reconstructing it by parsing the file’s metadata, which can +significantly reduce the time needed to open a file containing many pieces of cached metadata. The block +is pointed to by the @ref subsubsec_fmt4_dataobject_hdr_msg_mdci in the superblock extension. + +The block begins with a signature, version, flags, image data length, and entry count, followed by one +record for each metadata cache entry captured in the image, and ends with a checksum for the block. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Layout: Metadata Cache Image Block
    bytebytebytebyte
    Signature
    VersionFlagsThis space inserted only to align table nicely

    Image Data LengthL

    Number of Entries
    Image Entry \#0
    Image Entry \#1
    ...
    Image Entry \#N-1
    Checksum
    +\li Items marked with an ‘L’ in the above table are of the size specified in + “@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fields: Metadata Cache Image Block
    Field NameDescription
    SignatureThe ASCII character string “MDCI” is used to indicate the beginning of the + Metadata Cache Image Block. This gives file consistency checking utilities a better chance of + reconstructing a damaged file.
    VersionThis is the version number for the Metadata Cache Image Block and this document describes version 0.
    FlagsThis field holds flags for the block, as follows: + + + + + + + + + + + + + +
    BitsDescription
    0If set, adaptive cache resize status information follows the entries, below. The library does + not currently generate this information, so this bit is always 0.
    1-7Reserved (zero).
    Image Data LengthThis is the total length, in bytes, of the metadata cache image block, including this header, all + entries, and the trailing checksum. This is the same value stored in the “Metadata Cache Image + Block Size” field of the @ref subsubsec_fmt4_dataobject_hdr_msg_mdci that points to this + block.
    Number of EntriesThis is the number of metadata cache entry records, \a N, that follow the header. This value must be + greater than 0.
    Image Entry \#i (i = 0 to (N-1))These are the \a N metadata cache entry records captured in the image, one per entry that was + resident in the metadata cache at file close. The format of each entry is described below in + “Layout: Image Entry”.
    ChecksumThis is the checksum for the entire metadata cache image block, computed over the header and all + \a N entries that precede it (that is, over the first “Image Data Length” minus 4 bytes + of the block).
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Layout: Image Entry
    bytebytebytebyte
    Type IDFlagsRingAge
    Flush Dependency Child CountFlush Dependency Dirty Child Count
    Flush Dependency Parent CountThis space inserted only to align table nicely
    LRU Rank

    Entry OffsetO


    Entry LengthL

    Flush Dependency Parent AddressesO (0 or more, per “Flush Dependency + Parent Count”, above)
    Entry Image (variable size, per “Entry Length”, above)
    +\li Items marked with an ‘O’ in the above table are of the size specified in + “@ref FMT4SizeOfOffsetsV0 "Size of Offsets"” field in the superblock. +\li Items marked with an ‘L’ in the above table are of the size specified in + “@ref FMT4SizeOfLengthsV0 "Size of Lengths"” field in the superblock. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Fields: Image Entry
    Field NameDescription
    Type IDThis is an internal, implementation-defined identifier for the type of metadata cache client (for + example, superblock, object header, B-tree node, or fractal heap header) that owns this entry. The + library uses this value to determine how to reconstruct the entry in memory when the image is + loaded.
    FlagsThis field holds flags for the entry, as follows: + + + + + + + + + + + + + + + + + + + + + + + + + +
    BitsDescription
    0If set, the entry was dirty (not yet written to the file) at the time the image was + generated.
    1If set, the entry was on the cache’s LRU list, and “LRU Rank”, below, gives + its position on that list.
    2If set, the entry is the parent in one or more flush dependency relationships.
    3If set, the entry is a child in one or more flush dependency relationships, and the entry + record is followed by one or more “Flush Dependency Parent Address” fields, + below.
    4-7Reserved (zero).
    RingThis identifies the flush dependency “ring” the entry belongs to, which constrains the + order in which entries are flushed relative to entries in other rings (entries in an outer ring are + always flushed before entries in an inner ring), as follows: + + + + + + + + + + + + + + + + + + + + + + + + + +
    ValueDescription
    1User data (datasets and groups). Outermost ring; flushed first.
    2Raw data free-space manager.
    3Metadata free-space manager.
    4Superblock extension.
    5Superblock. Innermost ring; flushed last.
    AgeThis is the number of file opens and closes since the entry was last accessed by the application. The + library uses this value to decide whether to continue including the entry in future cache + images.
    Flush Dependency Child CountThis is the number of flush dependency children for which this entry is the parent. This is non-zero + only if bit 2 of “Flags”, above, is set.
    Flush Dependency Dirty Child CountThis is the number of flush dependency children, of the total given in “Flush Dependency Child + Count”, above, that were dirty at the time the image was generated.
    Flush Dependency Parent CountThis is the number of flush dependency parents for which this entry is a child, \a K. This is + non-zero only if bit 3 of “Flags”, above, is set.
    LRU RankIf bit 1 of “Flags”, above, is set, this is the entry’s position on the cache’s + LRU list (lower values are closer to the head of the list, i.e. more recently used). Otherwise, this + field has the value -1.
    Entry OffsetThis is the address in the file at which the entry’s metadata is (or will be) stored.
    Entry LengthThis is the length, in bytes, of the “Entry Image”, below.
    Flush Dependency Parent Address \#KThis is the file address of a flush dependency parent entry of this entry. There are as many of + these fields as given by “Flush Dependency Parent Count”, above.
    Entry ImageThis is a byte-for-byte copy of the entry’s serialized on-disk image, as it would otherwise + appear at “Entry Offset”, above, in the file (for example, the disk format of an object + header, B-tree node, or other cached metadata structure).
    + \section sec_fmt4_dataobject IV. Disk Format: Level 2 - Data Objects Data objects contain the “real” user-visible information in the file. These objects compose the scientific data and other information which are generally thought of as “data” by the @@ -4662,9 +4951,9 @@ Header messages are aligned on 8-byte boundaries for version 1 object headers. Size of Header Message \#n Data - This value specifies the number of bytes of header message data following the header message type and - length information for the current message. The size includes padding bytes to make the message a multiple - of eight bytes. + This value specifies the number of bytes of header message data following the header message type, + length, flags, and reserved fields for the current message. The size includes padding bytes to make + the message a multiple of eight bytes. Header Message \#n Flags @@ -4927,8 +5216,8 @@ or padding bytes inserted - they are stored packed together. Size of Header Message \#n Data This value specifies the number of bytes of header message data following the header message - type and length information for the current message. The size of messages in this version does - not include any padding bytes. + type, length, flags, and creation order (when present) fields for the current message. The size + of messages in this version does not include any padding bytes. Header Message \#n Flags @@ -10522,6 +10811,80 @@ described here (see also the description of @ref subsubsec_fmt4_dataobject_hdr_p +\subsubsection subsubsec_fmt4_dataobject_hdr_msg_mdci IV.A.3.y. The Metadata Cache Image Message + + + + + + + + + + + + + + + + + + + + +
    Header Message Name: Metadata Cache Image
    Header Message Type: 0x0018
    Length: Fixed
    Status: Optional; may not be repeated.
    Description:This message records the location and size of a metadata cache image block written to the file, when + generation of a metadata cache image is enabled for the file (see #H5Pset_mdc_image_config). The + metadata cache image block is a serialized copy of the contents of the metadata cache at file close, + which allows the library to reconstruct the cache directly from this block the next time the file is + opened, instead of reconstructing it by parsing the file's metadata. This can significantly reduce the + time needed to open a file containing many pieces of cached metadata.
    + The message is written to the superblock extension only when the file is closed with a metadata + cache image pending, and is removed from the superblock extension once the library has read the + referenced block back in at the next file open when the file is opened for read/write access, at + which point the file space it occupied is reclaimed. If the file is opened read-only, the message + and image block are left unchanged. See @ref subsec_fmt4_infra_mdci for the format of the metadata + cache image block itself.
    Format of Data: See the tables below.
    + + + + + + + + + + + + + + + + + + + +
    Layout: Metadata Cache Image
    bytebytebytebyte
    VersionThis space inserted only to align table nicely

    Metadata Cache Image Block AddressO


    Metadata Cache Image Block SizeL

    + + + + + + + + + + + + + + + + + + + +
    Fields: Metadata Cache Image
    Field NameDescription
    VersionThe version number for this message. This document describes version 0.
    Metadata Cache Image Block AddressThis is the address in the file of the metadata cache image block.
    Metadata Cache Image Block SizeThis is the size, in bytes, of the metadata cache image block.
    + \subsection subsec_fmt4_dataobject_storage IV.B. Disk Format: Level 2B - Data Object Data Storage The data for an object is stored separately from the header information in the file and may not actually be located in the HDF5 file itself if the header indicates that the data is stored externally. The