mirrors/hdf5
2
0
Fork 0
mirror of https://github.com/HDFGroup/hdf5.git synced 2026-09-25 04:09:44 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
de12d58bb5ec52c5583696b7d0f9c809cef5a7db
hdf5/develop/_comp_t_s.html
T
byrnHDF de12d58bb5 deploy: 53ef614d9e
2026-04-21 01:39:53 +00:00

558 lines
72 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=11"/>
<meta name="generator" content="Doxygen 1.16.1"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>HDF5: HDF5 Compression Troubleshooting</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript" src="cookie.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<script type="text/javascript">
$(function() { init_search(); });
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="hdf5doxy.css" rel="stylesheet" type="text/css"/>
<link href="doxygen-awesome.css" rel="stylesheet" type="text/css"/>
<link href="hdf5doxy.css" rel="stylesheet" type="text/css">
<script type="text/javascript" src="hdf5_navtree_hacks.js"></script>
<div style="background:#FFDDDD;font-size:120%;text-align:center;margin:0;padding:5px">Help us improve by taking our short survey: <a href="https://www.hdfgroup.org/website-survey/">https://www.hdfgroup.org/website-survey/</a></div>
<!-- ... other metadata & script includes ... -->
<script type="text/javascript" src="doxygen-awesome-tabs.js"></script>
<script type="text/javascript">
DoxygenAwesomeTabs.init()
</script>
<!-- Google tag (gtag.js) -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-57FMZK4S7X"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-57FMZK4S7X');
</script>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr id="projectrow">
<td id="projectlogo"><img alt="Logo" src="HDFG-logo.png"/></td>
<td id="projectalign">
<div id="projectname">HDF5<span id="projectnumber">&#160;Last Updated on 2026-04-21</span>
</div>
<div id="projectbrief">The HDF5 Field Guide</div>
</td>
</tr>
</tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.16.1 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "search/",'.html');
</script>
<script type="text/javascript">
$(function() { codefold.init(); });
</script>
<div id="main-nav">
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main&#160;Page</span></a></li>
<li><a href="_getting_started.html"><span>Getting&#160;started</span></a></li>
<li><a href="_u_g.html"><span>User&#160;Guide</span></a></li>
<li><a href="_r_m.html"><span>Reference&#160;Manual</span></a></li>
<li><a href="_cookbook.html"><span>Cookbook</span></a></li>
<li><a href="_t_n.html"><span>Technical&#160;Notes</span></a></li>
<li><a href="_r_f_c.html"><span>RFCs</span></a></li>
<li><a href="_s_p_e_c.html"><span>Specifications</span></a></li>
<li><a href="_g_l_s.html"><span>Glossary</span></a></li>
<li><a href="_f_t_s.html"><span>Full-Text&#160;Search</span></a></li>
<li><a href="_about.html"><span>About</span></a></li>
<li>
<div id="MSearchBox" class="MSearchBoxInactive">
<span class="left">
<span id="MSearchSelect" class="search-icon" onmouseover="return searchBox.OnSearchSelectShow()" onmouseout="return searchBox.OnSearchSelectHide()"><span class="search-icon-dropdown"></span></span>
<input type="text" id="MSearchField" value="" placeholder="Search" accesskey="S"
onfocus="searchBox.OnSearchFieldFocus(true)"
onblur="searchBox.OnSearchFieldFocus(false)"
onkeyup="searchBox.OnSearchFieldChange(event)"/>
</span><span class="right">
<a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><div id="MSearchCloseImg" class="close-icon"></div></a>
</span>
</div>
</li>
</ul>
</div>
</div><!-- main-nav -->
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(function(){initNavTree('_comp_t_s.html','',''); });
</script>
<div id="container">
<div id="doc-content">
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
onmouseover="return searchBox.OnSearchSelectShow()"
onmouseout="return searchBox.OnSearchSelectHide()"
onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>
<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<div id="MSearchResults">
<div class="SRPage">
<div id="SRIndex">
<div id="SRResults"></div>
<div class="SRStatus" id="Loading">Loading...</div>
<div class="SRStatus" id="Searching">Searching...</div>
<div class="SRStatus" id="NoMatches">No Matches</div>
</div>
</div>
</div>
</div>
<div><div class="header">
<div class="headertitle"><div class="title">HDF5 Compression Troubleshooting </div></div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>Navigate back: <a class="el" href="index.html" title="notitle">Main</a> / <a class="el" href="_t_n.html" title="Technical Notes">Technical Notes</a> </p><hr />
<p>The purpose of this technical note is to help HDF5 users with troubleshooting problems with <a class="el" href="_h5_z__u_g.html" title="HDF5 Filters">HDF5 Filters</a>, especially with compression filters. The document assumes that the reader knows HDF5 basics and is aware of the compression feature in HDF5.</p>
<h1 class="doxsection"><a class="anchor" id="sec_compts_intro"></a>
Introduction</h1>
<p>One of the most powerful features of HDF5 is the ability to modify, or “filter,” data during I/O. Filters provided by the HDF5 Library, “predefined filters”, include several types of data compression, data shuffling and checksum. Users can implement their own “user-defined filters” and use them with the HDF5 Library.</p>
<p>By far the most common user-defined filters are ones that perform data compression. While the programming model and usage of the compression filters are straightforward, it is easy, especially for novice users, to overlook important details when implementing compression filters and to end up with data that is not modified as they would expect.</p>
<p>The purpose of this document is to describe how to diagnose situations where the data in a file is not compressed as expected.</p>
<h1 class="doxsection"><a class="anchor" id="sec_compts_over"></a>
An Overview of HDF5 Compression Troubleshooting</h1>
<p>Sometimes users may find that HDF5 data was not compressed in a file or that the compression ratio is very small. By themselves, these results do not mean that compression did not work or did not work well. These results suggest that something might have gone wrong when a compression filter was applied. How can users determine the true cause of the problem?</p>
<p>There are two major reasons why a filter did not produce the desired result: it was not applied, or it was not effective.</p>
<h4>The filter was not applied</h4>
<p>If a filter was not applied at all, then it was not included at compile time when the library was built or was not found at run time for dynamically loaded filters.</p>
<p>The absence or presence of HDF5 predefined filters can be confirmed by examining the installed HDF5 files or by using HDF5 API calls. The absence or presence of all filter types can be confirmed by running HDF5 command-line tools on the produced HDF5 files. See <a class="el" href="#sec_compts_notapp" title="If a Filter Was Not Applied">If a Filter Was Not Applied</a> for more information.</p>
<h4>The filter was applied but was not effective</h4>
<p>The effectiveness of compression filters is a complex matter and is only briefly covered this document. See <a class="el" href="#sec_compts_ineff" title="If a Compression Filter Was Not Effective">If a Compression Filter Was Not Effective</a> for more information. This section gives a short overview of the problem and provides an example in which the advantages of different compression filters and their combinations are shown.</p>
<h1 class="doxsection"><a class="anchor" id="sec_compts_notapp"></a>
If a Filter Was Not Applied</h1>
<p>This section discusses how it may happen that a compression filter is not available to an application and describes the behavior of the HDF5 Library in the absence of the filter. Then we walk through how to troubleshoot the problem by checking the HDF5 installation, by examining what an application can do at run time to see if a filter is available, and by using some HDF5 command line tools to see if a filter was applied.</p>
<p>Note that there are internal predefined filters: </p><table class="doxtable">
<tr>
<td><a class="el" href="_h5_zpublic_8h.html#a9e802e9612b3647e7d3ffe4ce3b8dcce">H5Z_FILTER_DEFLATE</a></td><td>The gzip compression, or deflation, filter</td></tr>
<tr>
<td><a class="el" href="_h5_zpublic_8h.html#a421d9941c68ebb776573baeb9aa77cd2">H5Z_FILTER_SZIP</a></td><td>The SZIP compressionfilter</td></tr>
<tr>
<td><a class="el" href="_h5_zpublic_8h.html#a8cc463fa1979bd4bfa0dd9aa6a41e49d">H5Z_FILTER_NBIT</a></td><td>The N-bit compression filter</td></tr>
<tr>
<td><a class="el" href="_h5_zpublic_8h.html#a745d2ccb4f7712ed78ef5e562e27d2ca">H5Z_FILTER_SCALEOFFSET</a></td><td>The scale-offset compression filter</td></tr>
<tr>
<td><a class="el" href="_h5_zpublic_8h.html#aa723f1a71601bf22c95620a490ecf1af">H5Z_FILTER_SHUFFLE</a></td><td>The shuffle algorithm filter</td></tr>
<tr>
<td><a class="el" href="_h5_zpublic_8h.html#a59ca894c9c2b99b1614b0c46a7407f1c">H5Z_FILTER_FLETCHER32</a></td><td>The Fletcher32 checksum, or error checking, filter</td></tr>
</table>
<p>These are enabled by default by both <b>configure</b> and <b>CMake</b> builds. While these filters can be disabled intentionally with the <b>configure</b> flag <code>–disablefilters</code>, disabling them is not recommended. The discussion and the examples in this document focus on compression filters, but everything said can be applied to other missing internal filters as well.</p>
<h2 class="doxsection"><a class="anchor" id="subsec_compts_notapp_miss"></a>
How the HDF5 Library Configuration May Miss a Compression Filter</h2>
<p>The HDF5 Library uses external libraries for data compression. The two predefined compression methods are <b>gzip3</b> and <b>szip</b> or <b>libaec</b>, and these can be requested at the HDF5 Library configuration time (compile time). User-defined compression filters and the corresponding libraries are usually linked with an application or provided as a dynamically loaded library.</p>
<p><b>Note</b> that the <b>libaec</b> library is a replacement for the original <b>szip</b> library. The <b>libaec</b> library is a freely available, open-source library that provides compression and decompression functionality and is compatible with the <b>szip</b> filter. The <b>libaec</b> library can be used as a drop-in replacement for the <b>szip</b>, but requires two libraries to be present on the system: <b>libaec.a(so,dylib,lib)</b> and <b>libsz.a(so,dylib,lib)</b>. Everywhere in this document, the term <b>szip</b> refers to the <b>szip</b> filter and the <b>libaec</b> library.</p>
<p><b>gzip</b> and <b>szip</b> require the <b>libz.a(so,dylib,lib)</b> and <b>libsz.a(so,dylib,lib)/libaec</b>.a(so,dylib,lib) libraries, respectively, to be present on the system and to be enabled during HDF5 configuration with this CMake configure command: </p><div class="fragment"><div class="line">cmake -G <span class="stringliteral">&quot;Unix Makefiles&quot;</span> -DHDF5_ENABLE_ZLIB_SUPPORT=ON -DHDF5_ENABLE_SZIP_SUPPORT=ON -D&lt;other flags&gt;</div>
</div><!-- fragment --><p> With CMake both libraries have to be explicitly enabled. The source code distribution’s <code>config/cmake/cacheinit.cmake</code> file will enable both filters along with setting other options. Users can overwrite the defaults by using -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=OFF with the “cmake –C” command. See the <a href="https://github.com/HDFGroup/hdf5/blob/develop/docs/INSTALL_CMake.md">INSTALL_CMake.md</a> file under the docs directory in the HDF5 source distribution.</p>
<p>If compression is not requested or found at configuration time, the compression method is not registered with the library and cannot be applied when data is written or read. For example, the <a class="el" href="_h5_t_o_o_l__r_p__u_g.html#sec_cltools_h5repack" title="h5repack">h5repack</a> tool will not be able to remove an <b>szip</b> compression filter from a dataset if the <b>szip</b> library was not configured into the library against which the tool was built. The next section discusses the behavior of the HDF5 Library in the absence of filters.</p>
<h2 class="doxsection"><a class="anchor" id="subsec_compts_notapp_behave"></a>
How Does the HDF5 Library Behave in the Absence of a Filter</h2>
<p>By design, the HDF5 Library allows applications to create and write datasets using filters that are not available at creation/write time. This feature makes it possible to create HDF5 files on one system and to write data on another system where the HDF5 Library is configured with or without the requested filter.</p>
<p>Let’ s recall the HDF5 programming model for enabling filters.</p>
<p>An HDF5 application uses one or more <code>H5Pset_&lt;filter&gt;</code> calls to configure a dataset’s filter pipeline at its creation time. The excerpt below shows how a <b>gzip</b> filter is added to a pipeline5 with <a class="el" href="group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d" title="Sets deflate (GNU gzip) compression method and compression level.">H5Pset_deflate</a>. </p><div class="fragment"><div class="line"><span class="comment">// Create the dataset creation property list, add the gzip</span></div>
<div class="line"><span class="comment">// compression filter and set the chunk size.</span></div>
<div class="line">dcpl = <a class="code hl_function" href="group___p_l_c_r.html#gaf1b11da01d4d45d788c45f8bc5f0cbfa">H5Pcreate</a> (<a class="code hl_define" href="_h5_ppublic_8h.html#afcd7f8186c404f3a1d768632eacba102">H5P_DATASET_CREATE</a>);</div>
<div class="line">status = <a class="code hl_function" href="group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d">H5Pset_deflate</a> (dcpl, 9);</div>
<div class="line">status = <a class="code hl_function" href="group___d_c_p_l.html#ga3584d592e377da3604b7604e266dcf5b">H5Pset_chunk</a> (dcpl, 2, chunk);</div>
<div class="line">dset = <a class="code hl_define" href="group___h5_d.html#ga0647ba4bbd26d5230cc07f3a5685b2cf">H5Dcreate</a> (file, DATASET,…, dcpl,…);</div>
<div class="ttc" id="a_h5_ppublic_8h_html_afcd7f8186c404f3a1d768632eacba102"><div class="ttname"><a href="_h5_ppublic_8h.html#afcd7f8186c404f3a1d768632eacba102">H5P_DATASET_CREATE</a></div><div class="ttdeci">#define H5P_DATASET_CREATE</div><div class="ttdef"><b>Definition</b> H5Ppublic.h:60</div></div>
<div class="ttc" id="agroup___d_c_p_l_html_ga3584d592e377da3604b7604e266dcf5b"><div class="ttname"><a href="group___d_c_p_l.html#ga3584d592e377da3604b7604e266dcf5b">H5Pset_chunk</a></div><div class="ttdeci">herr_t H5Pset_chunk(hid_t plist_id, int ndims, const hsize_t dim[])</div><div class="ttdoc">Sets the size of the chunks used to store a chunked layout dataset.</div></div>
<div class="ttc" id="agroup___d_c_p_l_html_gaf1f569bfc54552bdb9317d2b63318a0d"><div class="ttname"><a href="group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d">H5Pset_deflate</a></div><div class="ttdeci">herr_t H5Pset_deflate(hid_t plist_id, unsigned level)</div><div class="ttdoc">Sets deflate (GNU gzip) compression method and compression level.</div></div>
<div class="ttc" id="agroup___h5_d_html_ga0647ba4bbd26d5230cc07f3a5685b2cf"><div class="ttname"><a href="group___h5_d.html#ga0647ba4bbd26d5230cc07f3a5685b2cf">H5Dcreate</a></div><div class="ttdeci">#define H5Dcreate</div><div class="ttdef"><b>Definition</b> H5version.h:1208</div></div>
<div class="ttc" id="agroup___p_l_c_r_html_gaf1b11da01d4d45d788c45f8bc5f0cbfa"><div class="ttname"><a href="group___p_l_c_r.html#gaf1b11da01d4d45d788c45f8bc5f0cbfa">H5Pcreate</a></div><div class="ttdeci">hid_t H5Pcreate(hid_t cls_id)</div><div class="ttdoc">Creates a new property list as an instance of a property list class.</div></div>
</div><!-- fragment --><p> For all internal filters (<b>shuffle</b>, <b>fletcher32</b>, <b>scaleoffset</b>, and <b>nbit</b>) and the external <b>gzip</b> filter, the HDF5 Library does not check to see if the filter is registered when the corresponding <code>H5Pset_&lt;filter&gt;</code> function is called. The only exception to this rule is <a class="el" href="group___d_c_p_l.html#ga37de4b6071a94574cfab5cd6de9c3fc6" title="Sets up use of the SZIP compression filter.">H5Pset_szip</a> which will fail if szip was not configured in or is configured with a decoder only. Hence, in the example above, <a class="el" href="group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d" title="Sets deflate (GNU gzip) compression method and compression level.">H5Pset_deflate</a> will succeed. The specified filter will be added to the dataset’s filter pipeline and will be applied to any data written to this dataset.</p>
<p>When <code>H5Pset_&lt;filter&gt;</code> is called, a record for the filter is added to the dataset’s object header in the file, and information about the filter can be queried with the HDF5 APIs and displayed by HDF5 tools such as <a class="el" href="_h5_t_o_o_l__d_p__u_g.html#sec_cltools_h5dump" title="h5dump">h5dump</a>. The presence of filter information in a dataset’s header does not mean that the filter was actually applied to the dataset’s data, as will be explained later in this document. See <a class="el" href="#subsec_compts_notapp_tools" title="How to Use HDF5 Tools to Investigate Missing Compression Filters">How to Use HDF5 Tools to Investigate Missing Compression Filters</a> for more information on how to use <a class="el" href="_h5_t_o_o_l__l_s__u_g.html#sec_cltools_h5ls" title="h5ls">h5ls</a> and <a class="el" href="_h5_t_o_o_l__d_g__u_g.html#sec_cltools_h5debug" title="h5debug">h5debug</a> to determine if the filter was actually applied.</p>
<p>The success of further write operations to a dataset when filters are missing depends on the filter type.</p>
<p>By design, an HDF5 filter can be optional or required. This filter mode defines the behavior of the HDF5 Library during write operations. In the absence of an optional filter, <a class="el" href="group___h5_d.html#ga98f44998b67587662af8b0d8a0a75906" title="Writes raw data from a buffer to a dataset.">H5Dwrite</a> calls will succeed and data will be written to the file, bypassing the filter. A missing required filter will cause <a class="el" href="group___h5_d.html#ga98f44998b67587662af8b0d8a0a75906" title="Writes raw data from a buffer to a dataset.">H5Dwrite</a> calls to fail. Clearly, <a class="el" href="group___h5_d.html#ga8287d5a7be7b8e55ffeff68f7d26811c" title="Reads raw data from a dataset into a provided buffer.">H5Dread</a> calls will fail when filters that are needed to decode the data are missing.</p>
<p>The HDF5 Library has only one required internal filter, <b>Fletcher32</b> (checksum creation), and one required external filter, <b>szip</b>. As mentioned earlier, only the <b>szip</b> compression (<a class="el" href="group___d_c_p_l.html#ga37de4b6071a94574cfab5cd6de9c3fc6" title="Sets up use of the SZIP compression filter.">H5Pset_szip</a>) will flag the absence of the filter. If, despite the missing filter, an application goes on to create a dataset via <a class="el" href="group___h5_d.html#ga0647ba4bbd26d5230cc07f3a5685b2cf">H5Dcreate</a>, the call will succeed, but the <b>szip</b> filter will not be added to the filter pipeline. This behavior is different from all other filters that may not be present, but will be added to the filter pipeline and applied during I/O. See the <a class="el" href="#subsubsec_compts_notapp_need_api" title="Using HDF5 APIs">Using HDF5 APIs</a> section for more information on how to determine if a filter is available and to avoid writing data while the filter is missing.</p>
<p>Developers who create their own filters should use the <b>flags</b> parameter in <a class="el" href="group___o_c_p_l.html#gae7ac4c110e9ec3fcf38966928b96cda3" title="Adds a filter to the filter pipeline.">H5Pset_filter</a> to declare if the filter is optional or required. The filter type can be determined by calling <a class="el" href="_h5version_8h.html#a7e070dfec9cb3a3aaf9c188a987e6a15">H5Pget_filter</a> and checking the value of the <b>flags</b> parameter.</p>
<p>For more information on filter behavior in HDF5, see <a class="el" href="_h5_z__u_g.html" title="HDF5 Filters">HDF5 Filters</a>.</p>
<h2 class="doxsection"><a class="anchor" id="subsec_compts_notapp_need"></a>
How to Determine if the HDF5 Library was Configured with a Needed Compression Filter</h2>
<p>The previous section described how the HDF5 Library could be configured without certain compression filters and the resulting expected library behavior.</p>
<p>The following subsections explain how to determine if a compression method is configured in the HDF5 Library and how to avoid accessing data if the filter is missing.</p>
<h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_need_settins"></a>
Examine the hdf5lib.settings File</h3>
<p>To see how the library was configured and built, users should examine the <code>hdf5lib.settings</code> text file found in the lib directory of the HDF5 installation point and search for the lines that contain the <code>“I/O
filters”</code> string. The <code>hdf5lib.settings</code> file is automatically generated at configuration time when the HDF5 Library is built with <b>configure</b> on Unix or with <b>CMake</b> on Unix and Windows, and it should contain the following lines: </p><div class="fragment"><div class="line">I/O filters (external): deflate(zlib),szip(encoder)</div>
<div class="line">I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset</div>
</div><!-- fragment --><p>The same lines in the file generated by <b>CMake</b> look slightly different: </p><div class="fragment"><div class="line">I/O filters (external): DEFLATE ENCODE DECODE</div>
<div class="line">I/O filters (internal): SHUFFLE FLETCHER32 NBIT SCALEOFFSET</div>
</div><!-- fragment --><p><code>“ENCODE DECODE”</code> indicates that both the <b>szip</b> compression encoder and decoder are present. This inconsistency between configure and CMake generated files will be removed in a future release. These lines show the compression libraries configured with HDF5. Here is an example of the same output when external compression filters are absent: </p><div class="fragment"><div class="line">I/O filters (external):</div>
<div class="line">I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset</div>
</div><!-- fragment --><p>Depending on the values listed on the I/O filters (external) line, users will be able to tell if their HDF5 files are compressed appropriately. If <b>szip</b> is not included in the build, data files will not be compressed with <b>szip</b>. If <b>gzip</b> is not included in the build and is not installed on the system, then data files will not be compressed with <b>gzip</b>.</p>
<p>If the <code>hdf5lib.settings</code> file is not present on the system, then users can examine a public header file or the library binary file to find out if a filter is present, as is discussed in the next two sections.</p>
<h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_need_head"></a>
Examine the H5pubconf.h Header File</h3>
<p>To see if a filter is present, users can also inspect the HDF5 public header file installed under the include directory of the HDF5 installation point. If the compression and internal filters are present, the corresponding symbols will be defined as follows: </p><div class="fragment"><div class="line"><span class="comment">// Define if support for deflate (zlib) filter is enabled</span></div>
<div class="line"><span class="preprocessor">#define H5_HAVE_FILTER_DEFLATE 1</span></div>
<div class="line"><span class="comment">// Define if support for Fletcher32 checksum is enabled</span></div>
<div class="line"><span class="preprocessor">#define H5_HAVE_FILTER_FLETCHER32 1</span></div>
<div class="line"><span class="comment">// Define if support for nbit filter is enabled</span></div>
<div class="line"><span class="preprocessor">#define H5_HAVE_FILTER_NBIT 1</span></div>
<div class="line"><span class="comment">// Define if support for scaleoffset filter is enabled</span></div>
<div class="line"><span class="preprocessor">#define H5_HAVE_FILTER_SCALEOFFSET 1</span></div>
<div class="line"><span class="comment">// Define if support for shuffle filter is enabled</span></div>
<div class="line"><span class="preprocessor">#define H5_HAVE_FILTER_SHUFFLE 1</span></div>
<div class="line"><span class="comment">// Define if support for Szip filter is enabled</span></div>
<div class="line"><span class="preprocessor">#define H5_HAVE_FILTER_SZIP 1</span></div>
</div><!-- fragment --><p> If a compression or internal filter was not configured, the corresponding lines will be commented out as follows: </p><div class="fragment"><div class="line"><span class="comment">// Define if support for deflate (zlib) filter is enabled</span></div>
<div class="line"><span class="comment">// #undef H5_HAVE_FILTER_DEFLATE</span></div>
</div><!-- fragment --><h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_need_bin"></a>
Check the HDF5 Library’s Binary</h3>
<p>The HDF5 Library’s binary contains summary output similar to what is stored in the <code>hdf5lib.settings</code> file. Users can run the Unix <code>“strings”</code> command to get information about the configured filters: </p><div class="fragment"><div class="line">% strings libhdf5.a(so) | grep <span class="stringliteral">&quot;I/O filters (&quot;</span></div>
<div class="line"> I/O filters (external): deflate(zlib),szip(encoder)</div>
<div class="line"> I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset</div>
</div><!-- fragment --><p> When compression filters are not configured, the output of the command above will be: </p><div class="fragment"><div class="line">I/O filters (external):</div>
<div class="line">I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset</div>
</div><!-- fragment --><p> On Windows one can use the <code>dumpbin /all</code> command, and then view and search the output for strings like <code>DEFLATE</code>, <code>FLETCHER32</code>, <code>DECODE</code>, and <code>ENCODE</code>. </p><div class="fragment"><div class="line">…..</div>
<div class="line">10201860: 4E 0A 20 20 20 20 20 20 20 20 20 49 2F 4F 20 66 N. I/O f</div>
<div class="line">10201870: 69 6C 74 65 72 73 20 28 65 78 74 65 72 6E 61 6C ilters (external</div>
<div class="line">10201880: 29 3A 20 20 44 45 46 4C 41 54 45 20 44 45 43 4F ): DEFLATE DECO</div>
<div class="line">10201890: 44 45 20 45 4E 43 4F 44 45 0A 20 20 20 20 20 20 DE ENCODE.</div>
<div class="line">102018A0: 20 20 20 49 2F 4F 20 66 69 6C 74 65 72 73 20 28 I/O filters (</div>
<div class="line">102018B0: 69 6E 74 65 72 6E 61 6C 29 3A 20 20 53 48 55 46 internal): SHUF</div>
<div class="line">102018C0: 46 4C 45 20 46 4C 45 54 43 48 45 52 33 32 20 4E FLE FLETCHER32 N</div>
<div class="line">102018D0: 42 49 54 20 53 43 41 4C 45 4F 46 46 53 45 54 0A BIT SCALEOFFSET.</div>
</div><!-- fragment --><h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_need_script"></a>
Check the Compiler Script</h3>
<p>Developers can also use the compiler scripts such as <code>h5cc</code> to verify that a compression library is present and configured in. Use the <code>- show</code> option with any of the compilers scripts found in the bin subdirectory of the HDF5 installation directory. The presence of <code>–lsz</code> and <code>–lz</code> options among the linker flags will confirm that <b>szip</b> or <b>gzip</b> were compiled with the HDF5 Library. See the sample below </p><div class="fragment"><div class="line">$ h5cc -show</div>
<div class="line">gcc -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_BSD_SOURCE -L/mnt/hdf/packages/<a class="code hl_namespace" href="namespacehdf5.html">hdf5</a>/v1812/Linux64_2.6/standard/lib</div>
<div class="line"> /mnt/hdf/packages/<a class="code hl_namespace" href="namespacehdf5.html">hdf5</a>/v1812/Linux64_2.6/standard/lib/libhdf5_hl.a</div>
<div class="line"> /mnt/hdf/packages/<a class="code hl_namespace" href="namespacehdf5.html">hdf5</a>/v1812/Linux64_2.6/standard/lib/libhdf5.a -lsz -lz -lrt</div>
<div class="line"> -ldl -lm -Wl,-rpath -Wl,/mnt/hdf/packages/<a class="code hl_namespace" href="namespacehdf5.html">hdf5</a>/v1812/Linux64_2.6/standard/lib</div>
<div class="ttc" id="anamespacehdf5_html"><div class="ttname"><a href="namespacehdf5.html">hdf5</a></div><div class="ttdef"><b>Definition</b> HDF5.F90:26</div></div>
</div><!-- fragment --><h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_need_cmake"></a>
Examine the hdf5-config.cmake File</h3>
<p><b>CMake</b> users can check the <code>hdf5-config.cmake</code> file in the <b>CMake</b> installation directory. The file will indicate what options were used to configure the HDF5 Library. The variables in the "User Options" section can be used by developers programmatically to determine if a filter was configured in.</p>
<p>After using <code>find-package(HDF5)</code> <b>CMake</b> can test the setting of these variables as shown below: </p><div class="fragment"><div class="line">find_package (HDF5 NAMES <a class="code hl_namespace" href="namespacehdf5.html">hdf5</a> COMPONENTS C)</div>
<div class="line"> </div>
<div class="line"><span class="keywordflow">if</span> (HDF5_ENABLE_ZLIB_SUPPORT)</div>
<div class="line"> message(STATUS <span class="stringliteral">&quot;gzip filter is available.&quot;</span>)</div>
<div class="line"><span class="keywordflow">else</span>()</div>
<div class="line"> message(STATUS <span class="stringliteral">&quot;gzip filter is not available.&quot;</span>)</div>
<div class="line">endif()</div>
</div><!-- fragment --><h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_need_api"></a>
Using HDF5 APIs</h3>
<p>Applications can check filter availability at run time. In order to check the filter’s availability with the HDF5 Library, users should know the filter identifier (for example, <a class="el" href="_h5_zpublic_8h.html#a9e802e9612b3647e7d3ffe4ce3b8dcce">H5Z_FILTER_DEFLATE</a>) and call the <a class="el" href="group___h5_z.html#ga3594e10d70739ccda55ebb55b17b50ee" title="Determines whether a filter is available.">H5Zfilter_avail</a> function as shown in the example below. Use <a class="el" href="group___h5_z.html#ga9ef800ceec249c8819492545def9adba" title="Retrieves information about a filter.">H5Zget_filter_info</a> to determine if the filter is configured to decode data, to encode data, neither, or both. </p><div class="fragment"><div class="line"><span class="comment">// Check if gzip compression is available and can be used for both compression and decompression.</span></div>
<div class="line">avail = <a class="code hl_function" href="group___h5_z.html#ga3594e10d70739ccda55ebb55b17b50ee">H5Zfilter_avail</a>(<a class="code hl_define" href="_h5_zpublic_8h.html#a9e802e9612b3647e7d3ffe4ce3b8dcce">H5Z_FILTER_DEFLATE</a>);</div>
<div class="line"><span class="keywordflow">if</span> (!avail) {</div>
<div class="line"> printf (<span class="stringliteral">&quot;gzip filter not available.\n&quot;</span>);</div>
<div class="line"> <span class="keywordflow">return</span> 1;</div>
<div class="line">}</div>
<div class="line">status = <a class="code hl_function" href="group___h5_z.html#ga9ef800ceec249c8819492545def9adba">H5Zget_filter_info</a> (<a class="code hl_define" href="_h5_zpublic_8h.html#a9e802e9612b3647e7d3ffe4ce3b8dcce">H5Z_FILTER_DEFLATE</a>, &amp;filter_info);</div>
<div class="line"><span class="keywordflow">if</span> ( !(filter_info &amp; <a class="code hl_define" href="_h5_zpublic_8h.html#ac4ec01a86fdac6619c7c3c1fcf3bf86a">H5Z_FILTER_CONFIG_ENCODE_ENABLED</a>) ||</div>
<div class="line"> !(filter_info &amp; <a class="code hl_define" href="_h5_zpublic_8h.html#a4dead61ceb139a3f97505d6e52eb1b8a">H5Z_FILTER_CONFIG_DECODE_ENABLED</a>) ) {</div>
<div class="line"> printf (<span class="stringliteral">&quot;gzip filter not available for encoding and decoding.\n&quot;</span>);</div>
<div class="line"> <span class="keywordflow">return</span> 1;</div>
<div class="line">}</div>
<div class="ttc" id="a_h5_zpublic_8h_html_a4dead61ceb139a3f97505d6e52eb1b8a"><div class="ttname"><a href="_h5_zpublic_8h.html#a4dead61ceb139a3f97505d6e52eb1b8a">H5Z_FILTER_CONFIG_DECODE_ENABLED</a></div><div class="ttdeci">#define H5Z_FILTER_CONFIG_DECODE_ENABLED</div><div class="ttdef"><b>Definition</b> H5Zpublic.h:273</div></div>
<div class="ttc" id="a_h5_zpublic_8h_html_a9e802e9612b3647e7d3ffe4ce3b8dcce"><div class="ttname"><a href="_h5_zpublic_8h.html#a9e802e9612b3647e7d3ffe4ce3b8dcce">H5Z_FILTER_DEFLATE</a></div><div class="ttdeci">#define H5Z_FILTER_DEFLATE</div><div class="ttdef"><b>Definition</b> H5Zpublic.h:66</div></div>
<div class="ttc" id="a_h5_zpublic_8h_html_ac4ec01a86fdac6619c7c3c1fcf3bf86a"><div class="ttname"><a href="_h5_zpublic_8h.html#ac4ec01a86fdac6619c7c3c1fcf3bf86a">H5Z_FILTER_CONFIG_ENCODE_ENABLED</a></div><div class="ttdeci">#define H5Z_FILTER_CONFIG_ENCODE_ENABLED</div><div class="ttdef"><b>Definition</b> H5Zpublic.h:270</div></div>
<div class="ttc" id="agroup___h5_z_html_ga3594e10d70739ccda55ebb55b17b50ee"><div class="ttname"><a href="group___h5_z.html#ga3594e10d70739ccda55ebb55b17b50ee">H5Zfilter_avail</a></div><div class="ttdeci">htri_t H5Zfilter_avail(H5Z_filter_t id)</div><div class="ttdoc">Determines whether a filter is available.</div></div>
<div class="ttc" id="agroup___h5_z_html_ga9ef800ceec249c8819492545def9adba"><div class="ttname"><a href="group___h5_z.html#ga9ef800ceec249c8819492545def9adba">H5Zget_filter_info</a></div><div class="ttdeci">herr_t H5Zget_filter_info(H5Z_filter_t filter, unsigned int *filter_config_flags)</div><div class="ttdoc">Retrieves information about a filter.</div></div>
</div><!-- fragment --><p><a class="el" href="group___h5_z.html#ga3594e10d70739ccda55ebb55b17b50ee" title="Determines whether a filter is available.">H5Zfilter_avail</a> can be used to find filters that are registered with the library or are available via dynamically loaded libraries. For more information, see <a class="el" href="_h5_d__u_g.html#subsubsec_dataset_filters_dyn" title="Using Dynamically-Loadable Filters">Using Dynamically-Loadable Filters</a>.</p>
<p>Currently there is no HDF5 API call to retrieve a list of all of the registered or dynamically loaded filters. The default installation directories for HDF5 dynamically loaded filters are <code>/usr/local/hdf5/lib/plugin</code> on Unix and <code>ALLUSERSPROFILE%\<a class="el" href="namespacehdf5.html">hdf5</a>\lib\plugin</code> on Windows. Users can also check to see if the environment variable <code>HDF5_PLUGIN_PATH</code> is set on the system and refers to a directory with available plugins.</p>
<h2 class="doxsection"><a class="anchor" id="subsec_compts_notapp_tools"></a>
How to Use HDF5 Tools to Investigate Missing Compression Filters</h2>
<p>In this section, we will use the <a class="el" href="_h5_t_o_o_l__d_p__u_g.html#sec_cltools_h5dump" title="h5dump">h5dump</a>, <a class="el" href="_h5_t_o_o_l__l_s__u_g.html#sec_cltools_h5ls" title="h5ls">h5ls</a>, and <a class="el" href="_h5_t_o_o_l__d_g__u_g.html#sec_cltools_h5debug" title="h5debug">h5debug</a> command-line utilities to see if a file was created with an HDF5 Library that did or did not have a compression filter configured in. For more information on these tools, see the <a class="el" href="_u_g.html#sec_cltools" title="Command Line Tools for HDF5 Files">Command Line Tools for HDF5 Files</a> page in the <a class="el" href="_u_g.html" title="HDF5 User Guide">HDF5 User Guide</a>.</p>
<h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_tools_dump"></a>
How to Use h5dump to Examine Files with Compressed Data</h3>
<p>The <a class="el" href="_h5_t_o_o_l__d_p__u_g.html#sec_cltools_h5dump" title="h5dump">h5dump</a> command-line tool can be used to see if a file uses a compression filter. The tool has two flags that will limit the output: the <code>–p</code> flag causes dataset properties including compression filters to be displayed, and the <code>–H</code> flag is used to suppress the output of data. The program provided in the <a class="el" href="_u_g.html#sec_cltools" title="Command Line Tools for HDF5 Files">Command Line Tools for HDF5 Files</a> section creates a file called <code>h5ex_d_gzip.h5</code>. The output of <a class="el" href="_h5_t_o_o_l__d_p__u_g.html#sec_cltools_h5dump" title="h5dump">h5dump</a> shows that the <b>gzip</b> compression filter set to level 9 was added to the DS1 dataset filter pipeline at creation time.</p>
<div class="fragment"><div class="line">$ <a class="code hl_namespace" href="namespacehdf5.html">hdf5</a>/bin/h5dump -p -H *.h5</div>
<div class="line">HDF5 <span class="stringliteral">&quot;h5ex_d_gzip.h5&quot;</span> {</div>
<div class="line">GROUP <span class="stringliteral">&quot;/&quot;</span> {</div>
<div class="line"> DATASET <span class="stringliteral">&quot;DS1&quot;</span> {</div>
<div class="line"> DATATYPE <a class="code hl_define" href="group___p_d_t_s_t_d.html#ga8db8c9c2bcc457f9f8526c8fcb81218b">H5T_STD_I32LE</a></div>
<div class="line"> DATASPACE SIMPLE { ( 32, 64 ) / ( 32, 64 ) }</div>
<div class="line"> STORAGE_LAYOUT {</div>
<div class="line"> CHUNKED ( 5, 9 )</div>
<div class="line"> SIZE 5018 (1.633:1 COMPRESSION)</div>
<div class="line"> }</div>
<div class="line"> FILTERS {</div>
<div class="line"> COMPRESSION DEFLATE { LEVEL 9 }</div>
<div class="line"> }</div>
<div class="line"> FILLVALUE {</div>
<div class="line"> FILL_TIME <a class="code hl_enumvalue" href="_h5_dpublic_8h.html#aa39293626c4e68dd28b06c0dc84bde4aa85b225308b0a277c4dd6fed7ee465a72">H5D_FILL_TIME_IFSET</a></div>
<div class="line"> VALUE 0</div>
<div class="line"> }</div>
<div class="line"> ALLOCATION_TIME {</div>
<div class="line"> <a class="code hl_enumvalue" href="_h5_dpublic_8h.html#aab70b464cf3c5fc931dce0c4fe98b3d2ac898a96931fd3402d9e5646690c77636">H5D_ALLOC_TIME_INCR</a></div>
<div class="line"> }</div>
<div class="line"> }</div>
<div class="line">}</div>
<div class="line">}</div>
<div class="ttc" id="a_h5_dpublic_8h_html_aa39293626c4e68dd28b06c0dc84bde4aa85b225308b0a277c4dd6fed7ee465a72"><div class="ttname"><a href="_h5_dpublic_8h.html#aa39293626c4e68dd28b06c0dc84bde4aa85b225308b0a277c4dd6fed7ee465a72">H5D_FILL_TIME_IFSET</a></div><div class="ttdeci">@ H5D_FILL_TIME_IFSET</div><div class="ttdef"><b>Definition</b> H5Dpublic.h:119</div></div>
<div class="ttc" id="a_h5_dpublic_8h_html_aab70b464cf3c5fc931dce0c4fe98b3d2ac898a96931fd3402d9e5646690c77636"><div class="ttname"><a href="_h5_dpublic_8h.html#aab70b464cf3c5fc931dce0c4fe98b3d2ac898a96931fd3402d9e5646690c77636">H5D_ALLOC_TIME_INCR</a></div><div class="ttdeci">@ H5D_ALLOC_TIME_INCR</div><div class="ttdef"><b>Definition</b> H5Dpublic.h:94</div></div>
<div class="ttc" id="agroup___p_d_t_s_t_d_html_ga8db8c9c2bcc457f9f8526c8fcb81218b"><div class="ttname"><a href="group___p_d_t_s_t_d.html#ga8db8c9c2bcc457f9f8526c8fcb81218b">H5T_STD_I32LE</a></div><div class="ttdeci">#define H5T_STD_I32LE</div><div class="ttdef"><b>Definition</b> H5Tpublic.h:471</div></div>
</div><!-- fragment --><p>The output also shows a compression ratio defined as (original size)/(storage size). The size of the stored data is 5018 bytes vs. 8192 bytes of uncompressed data, a ratio of 1.663. This shows that the filter was successfully applied.</p>
<p>Now let’s look at what happens when the same program is linked against an HDF5 Library that was not configured with the <b>gzip</b> library.</p>
<p>Notice that some chunks are only partially filled. 56 chunks (7 along the first dimension and 8 along the second dimension) are required to store the data. Since no compression was applied, each chunk has size <code>5x9x4 = 180</code> bytes, resulting in a total storage size of 10,080 bytes. With an original size of 8192 bytes, the compression ratio is 0.813 (in other words, less than 1) and visible in the output below. </p><div class="fragment"><div class="line">$ <a class="code hl_namespace" href="namespacehdf5.html">hdf5</a>/bin/h5dump -p -H *.h5</div>
<div class="line">HDF5 <span class="stringliteral">&quot;h5ex_d_gzip.h5&quot;</span> {</div>
<div class="line">GROUP <span class="stringliteral">&quot;/&quot;</span> {</div>
<div class="line"> DATASET <span class="stringliteral">&quot;DS1&quot;</span> {</div>
<div class="line"> DATATYPE <a class="code hl_define" href="group___p_d_t_s_t_d.html#ga8db8c9c2bcc457f9f8526c8fcb81218b">H5T_STD_I32LE</a></div>
<div class="line"> DATASPACE SIMPLE { ( 32, 64 ) / ( 32, 64 ) }</div>
<div class="line"> STORAGE_LAYOUT {</div>
<div class="line"> CHUNKED ( 5, 9 )</div>
<div class="line"> SIZE 10080 (0.813:1 COMPRESSION)</div>
<div class="line"> }</div>
<div class="line"> FILTERS {</div>
<div class="line"> COMPRESSION DEFLATE { LEVEL 9 }</div>
<div class="line"> }</div>
<div class="line"> FILLVALUE {</div>
<div class="line"> FILL_TIME <a class="code hl_enumvalue" href="_h5_dpublic_8h.html#aa39293626c4e68dd28b06c0dc84bde4aa85b225308b0a277c4dd6fed7ee465a72">H5D_FILL_TIME_IFSET</a></div>
<div class="line"> VALUE 0</div>
<div class="line"> }</div>
<div class="line"> ALLOCATION_TIME {</div>
<div class="line"> <a class="code hl_enumvalue" href="_h5_dpublic_8h.html#aab70b464cf3c5fc931dce0c4fe98b3d2ac898a96931fd3402d9e5646690c77636">H5D_ALLOC_TIME_INCR</a></div>
<div class="line"> }</div>
<div class="line"> }</div>
<div class="line">}</div>
<div class="line">}</div>
</div><!-- fragment --><p> As discussed in the <a class="el" href="#subsec_compts_notapp_behave" title="How Does the HDF5 Library Behave in the Absence of a Filter">How Does the HDF5 Library Behave in the Absence of a Filter</a>, the presence of a filter in an object’s filter pipeline <b>does not imply</b> that it will be applied unconditionally when data is written.</p>
<p>If the compression ratio is less than 1, compression is not applied. If it is 1, and compression is shown by <a class="el" href="_h5_t_o_o_l__d_p__u_g.html#sec_cltools_h5dump" title="h5dump">h5dump</a>, more investigation is needed; this will be discussed in the next section.</p>
<h3 class="doxsection"><a class="anchor" id="subsubsec_compts_notapp_tools_debug"></a>
How to Use h5ls and h5debug to Find a Missing Compression Filter</h3>
<p>Filters operate on chunked datasets. A filter may be ineffective for one chunk (for example, the compressed data is bigger than the original data), and succeed on another. How can users discern if a filter is missing or just ineffective (and as a result non-compressed data was written)? The <a class="el" href="_h5_t_o_o_l__l_s__u_g.html#sec_cltools_h5ls" title="h5ls">h5ls</a> and <a class="el" href="_h5_t_o_o_l__d_g__u_g.html#sec_cltools_h5debug" title="h5debug">h5debug</a> command-line tools can be used to investigate the issue.</p>
<p>First, let’s take a look at what kind of information <a class="el" href="_h5_t_o_o_l__l_s__u_g.html#sec_cltools_h5ls" title="h5ls">h5ls</a> displays about the dataset DS1 in our example file, which was written with an HDF5 library that has the <b>deflate</b> filter configured in: </p><div class="fragment"><div class="line">$ h5ls -vr h5ex_d_gzip.h5</div>
<div class="line">Opened <span class="stringliteral">&quot;h5ex_d_gzip.h5&quot;</span> with sec2 driver.</div>
<div class="line">/ Group</div>
<div class="line"> Location: 1:96</div>
<div class="line"> Links: 1</div>
<div class="line">/DS1 Dataset {32/32, 64/64}</div>
<div class="line"> Location: 1:800</div>
<div class="line"> Links: 1</div>
<div class="line"> Chunks: {5, 9} 180 bytes</div>
<div class="line"> Storage: 8192 logical bytes, 5018 allocated bytes, 163.25% utilization</div>
<div class="line"> Filter-0: deflate-1 OPT {9}</div>
<div class="line"> Type: native <span class="keywordtype">int</span></div>
</div><!-- fragment --><p>We see output similar to <a class="el" href="_h5_t_o_o_l__d_p__u_g.html#sec_cltools_h5dump" title="h5dump">h5dump</a> output with the compression ratio at 163%.</p>
<p>Now let’s compare this output with another dataset <b>DS1</b>, but this time the dataset was written with a program linked against an HDF5 library without the <b>gzip</b> filter present. </p><div class="fragment"><div class="line">$ h5ls -vr h5ex_d_gzip.h5</div>
<div class="line">Opened <span class="stringliteral">&quot;h5ex_d_gzip.h5&quot;</span> with sec2 driver.</div>
<div class="line">/ Group</div>
<div class="line"> Location: 1:96</div>
<div class="line"> Links: 1</div>
<div class="line">/DS1 Dataset {32/32, 64/64}</div>
<div class="line"> Location: 1:800</div>
<div class="line"> Links: 1</div>
<div class="line"> Chunks: {5, 9} 180 bytes</div>
<div class="line"> Storage: 8192 logical bytes, 10080 allocated bytes, 81.27% utilization</div>
<div class="line"> Filter-0: deflate-1 OPT {9}</div>
<div class="line"> Type: native <span class="keywordtype">int</span></div>
</div><!-- fragment --><p>The <a class="el" href="_h5_t_o_o_l__l_s__u_g.html#sec_cltools_h5ls" title="h5ls">h5ls</a> output above shows that the <b>gzip</b> filter was added to the filter pipeline of the dataset <b>DS1</b>. It also shows that the compression ratio is less than 1. We can confirm by using <a class="el" href="_h5_t_o_o_l__d_g__u_g.html#sec_cltools_h5debug" title="h5debug">h5debug</a> that the filter was not applied at all, and, as a result of the missing filter, the individual chunks were not compressed.</p>
<p>From the <a class="el" href="_h5_t_o_o_l__l_s__u_g.html#sec_cltools_h5ls" title="h5ls">h5ls</a> output we know that the dataset object header is located at address 800. We retrieve the dataset object header at address 800 and search the layout message for the address of the chunk index B-tree as shown in the excerpt of the <a class="el" href="_h5_t_o_o_l__d_g__u_g.html#sec_cltools_h5debug" title="h5debug">h5debug</a> output below: </p><div class="fragment"><div class="line">$ h5debug h5ex_d_gzip.h5 800</div>
<div class="line">Reading signature at address 800 (rel)</div>
<div class="line">Object Header...</div>
<div class="line">…..</div>
<div class="line">Message 4...</div>
<div class="line"> Message ID (sequence number): 0x0008 `layout<span class="stringliteral">&#39; (0)</span></div>
<div class="line"><span class="stringliteral"> Dirty: FALSE</span></div>
<div class="line"><span class="stringliteral"> Message flags: &lt;C&gt;</span></div>
<div class="line"><span class="stringliteral"> Chunk number: 0</span></div>
<div class="line"><span class="stringliteral"> Raw message data (offset, size) in chunk: (144, 24) bytes</span></div>
<div class="line"><span class="stringliteral"> Message Information:</span></div>
<div class="line"><span class="stringliteral"> Version: 3</span></div>
<div class="line"><span class="stringliteral"> Type: Chunked</span></div>
<div class="line"><span class="stringliteral"> Number of dimensions: 3</span></div>
<div class="line"><span class="stringliteral"> Size: {5, 9, 4}</span></div>
<div class="line"><span class="stringliteral"> Index Type: v1 B-tree</span></div>
<div class="line"><span class="stringliteral"> B-tree address: 1400</span></div>
</div><!-- fragment --><p>Now we can retrieve the B-tree information: </p><div class="fragment"><div class="line">$ h5debug h5ex_d_gzip.h5 1400 3</div>
<div class="line">Reading signature at address 1400 (rel)</div>
<div class="line">Tree type ID: H5B_CHUNK_ID</div>
<div class="line">Size of node: 2616</div>
<div class="line">Size of raw (disk) key: 32</div>
<div class="line">Dirty flag: False</div>
<div class="line">Level: 0</div>
<div class="line">Address of left sibling: UNDEF</div>
<div class="line">Address of right sibling: UNDEF</div>
<div class="line">Number of children (max): 56 (64)</div>
<div class="line">Child 0...</div>
<div class="line"> Address: 4016</div>
<div class="line"> Left Key:</div>
<div class="line"> Chunk size: 180 bytes</div>
<div class="line"> Filter mask: 0x00000001</div>
<div class="line"> Logical offset: {0, 0, 0}</div>
<div class="line"> Right Key:</div>
<div class="line"> Chunk size: 180 bytes</div>
<div class="line"> Filter mask: 0x00000001</div>
<div class="line"> Logical offset: {0, 9, 0}</div>
<div class="line">Child 1...</div>
<div class="line"> Address: 4196</div>
<div class="line"> Left Key:</div>
<div class="line"> Chunk size: 180 bytes</div>
</div><!-- fragment --> <ul>
<li>Users have to supply the chunk rank. According to the HDF5 <a class="el" href="_s_p_e_c.html#sec_spec_ff" title="File Format">File Format</a> Specification, this is the dataset rank plus 1; in other words, 3.</li>
</ul>
<p>We see that the size of each chunk is 180 bytes: in other words, compression was not successful. The filter mask value 0x00000001 indicates that filter was not applied. For more information on the filter mask, see the <a class="el" href="_f_m_t3.html#subsubsec_fmt3_infra_btrees_v1" title="III.A.1. Disk Format: Level 1A1 - Version 1 B-trees">III.A.1. Disk Format: Level 1A1 - Version 1 B-trees</a> section in the HDF5 <a class="el" href="_s_p_e_c.html#sec_spec_ff" title="File Format">File Format</a> Specification.</p>
<h2 class="doxsection"><a class="anchor" id="subsec_compts_notapp_ex"></a>
Example Program</h2>
<p>The example program used to create the file discussed in this document is a modified version of the program available at <a href="https://github.com/HDFGroup/hdf5/blob/develop/HDF5Examples/C/H5D/h5ex_d_gzip.c">h5ex_d_gzip.c</a>. It was modified to have chunk dimensions not be factors of the dataset dimensions. Chunk dimensions were chosen for demonstration purposes only and are not recommended for real applications. </p><div class="fragment"><div class="line"><span class="preprocessor">#include &lt;stdlib.h&gt;</span></div>
<div class="line"> </div>
<div class="line"><span class="preprocessor">#define FILE &quot;h5ex_d_gzip.h5&quot;</span></div>
<div class="line"><span class="preprocessor">#define DATASET &quot;DS1&quot;</span></div>
<div class="line"><span class="preprocessor">#define DIM0 32</span></div>
<div class="line"><span class="preprocessor">#define DIM1 64</span></div>
<div class="line"><span class="preprocessor">#define CHUNK0 5</span></div>
<div class="line"><span class="preprocessor">#define CHUNK1 9</span></div>
<div class="line"> </div>
<div class="line"><span class="keywordtype">int</span> main (<span class="keywordtype">void</span>)</div>
<div class="line">{</div>
<div class="line"> <a class="code hl_typedef" href="_h5_ipublic_8h.html#a0045db7ff9c22ad35db6ae91662e1943">hid_t</a> file, space, dset, dcpl; <span class="comment">// Handles</span></div>
<div class="line"> <a class="code hl_typedef" href="_h5public_8h.html#a3b079ecf932a5c599499cf7e298af160">herr_t</a> status;</div>
<div class="line"> <a class="code hl_typedef" href="_h5public_8h.html#aa8f6c28736dbd0f18388c67911d38aca">htri_t</a> avail;</div>
<div class="line"> <a class="code hl_typedef" href="_h5_zpublic_8h.html#afae8461c70d47e63be2163af23362237">H5Z_filter_t</a> filter_type;</div>
<div class="line"> <a class="code hl_typedef" href="_h5public_8h.html#a7f81cce70fb546af88da24d9285d3c1c">hsize_t</a> dims[2] = {DIM0, DIM1}, chunk[2] = {CHUNK0, CHUNK1};</div>
<div class="line"> <span class="keywordtype">size_t</span> nelmts;</div>
<div class="line"> <span class="keywordtype">unsigned</span> <span class="keywordtype">int</span> flags, filter_info;</div>
<div class="line"> <span class="keywordtype">int</span> wdata[DIM0][DIM1], <span class="comment">// Write buffer</span></div>
<div class="line"> rdata[DIM0][DIM1], <span class="comment">// Read buffer</span></div>
<div class="line"> max, i, j;</div>
<div class="line"> </div>
<div class="line"> <span class="comment">// Initialize data.</span></div>
<div class="line"> <span class="keywordflow">for</span> (i=0; i&lt;DIM0; i++)</div>
<div class="line"> <span class="keywordflow">for</span> (j=0; j&lt;DIM1; j++)</div>
<div class="line"> wdata[i][j] = i * j - j;</div>
<div class="line"> </div>
<div class="line"> <span class="comment">// Create a new file using the default properties.</span></div>
<div class="line"> file = <a class="code hl_function" href="group___h5_f.html#gae64b51ee9ac0781bc4ccc599d98387f4">H5Fcreate</a> (FILE, <a class="code hl_define" href="_h5_fpublic_8h.html#a5a2d6726f9ad8d2bca8df2b817e5ad6a">H5F_ACC_TRUNC</a>, <a class="code hl_define" href="_h5_ppublic_8h.html#afa85e97bfbf9bf1c58e39263846c568f">H5P_DEFAULT</a>, <a class="code hl_define" href="_h5_ppublic_8h.html#afa85e97bfbf9bf1c58e39263846c568f">H5P_DEFAULT</a>);</div>
<div class="line"> </div>
<div class="line"> <span class="comment">// Create dataspace. Setting maximum size to NULL sets the maximum</span></div>
<div class="line"> <span class="comment">// size to be the current size.</span></div>
<div class="line"> space = <a class="code hl_function" href="group___h5_s.html#ga8e35eea5738b4805856eac7d595254ae">H5Screate_simple</a> (2, dims, NULL);</div>
<div class="line"> </div>
<div class="line"> <span class="comment">// Create the dataset creation property list, add the gzip</span></div>
<div class="line"> <span class="comment">// compression filter and set the chunk size.</span></div>
<div class="line"> dcpl = <a class="code hl_function" href="group___p_l_c_r.html#gaf1b11da01d4d45d788c45f8bc5f0cbfa">H5Pcreate</a> (<a class="code hl_define" href="_h5_ppublic_8h.html#afcd7f8186c404f3a1d768632eacba102">H5P_DATASET_CREATE</a>);</div>
<div class="line"> status = <a class="code hl_function" href="group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d">H5Pset_deflate</a> (dcpl, 9);</div>
<div class="line"> status = <a class="code hl_function" href="group___d_c_p_l.html#ga3584d592e377da3604b7604e266dcf5b">H5Pset_chunk</a> (dcpl, 2, chunk);</div>
<div class="line"> </div>
<div class="line"> <span class="comment">// Create the dataset.</span></div>
<div class="line"> dset = <a class="code hl_define" href="group___h5_d.html#ga0647ba4bbd26d5230cc07f3a5685b2cf">H5Dcreate</a> (file, DATASET, <a class="code hl_define" href="group___p_d_t_s_t_d.html#ga8db8c9c2bcc457f9f8526c8fcb81218b">H5T_STD_I32LE</a>, space, <a class="code hl_define" href="_h5_ppublic_8h.html#afa85e97bfbf9bf1c58e39263846c568f">H5P_DEFAULT</a>, dcpl, <a class="code hl_define" href="_h5_ppublic_8h.html#afa85e97bfbf9bf1c58e39263846c568f">H5P_DEFAULT</a>);</div>
<div class="line"> </div>
<div class="line"> <span class="comment">// Write the data to the dataset.</span></div>
<div class="line"> status = <a class="code hl_function" href="group___h5_d.html#ga98f44998b67587662af8b0d8a0a75906">H5Dwrite</a> (dset, <a class="code hl_define" href="group___p_d_t_n_a_t.html#ga3cf93ffc6782be68070ef8e00f219ec2">H5T_NATIVE_INT</a>, <a class="code hl_define" href="_h5_spublic_8h.html#a5f96eeee84b987f18470737f85af0484">H5S_ALL</a>, <a class="code hl_define" href="_h5_spublic_8h.html#a5f96eeee84b987f18470737f85af0484">H5S_ALL</a>, <a class="code hl_define" href="_h5_ppublic_8h.html#afa85e97bfbf9bf1c58e39263846c568f">H5P_DEFAULT</a>, wdata[0]);</div>
<div class="line"> </div>
<div class="line"> <span class="comment">// Close and release resources.</span></div>
<div class="line"> status = <a class="code hl_function" href="group___p_l_c_r.html#ga5dce61149211d3ef319452aa598887fb">H5Pclose</a> (dcpl);</div>
<div class="line"> status = <a class="code hl_function" href="group___h5_d.html#gae47c3f38db49db127faf221624c30609">H5Dclose</a> (dset);</div>
<div class="line"> status = <a class="code hl_function" href="group___h5_s.html#ga2b53128a39c8f104c1c9c2a91590fcc1">H5Sclose</a> (space);</div>
<div class="line"> status = <a class="code hl_function" href="group___h5_f.html#gac55cd91d80822e4f8c2a7f04ea71b124">H5Fclose</a> (file);</div>
<div class="line"> <span class="keywordflow">return</span> 0;</div>
<div class="line">}</div>
<div class="ttc" id="a_h5_fpublic_8h_html_a5a2d6726f9ad8d2bca8df2b817e5ad6a"><div class="ttname"><a href="_h5_fpublic_8h.html#a5a2d6726f9ad8d2bca8df2b817e5ad6a">H5F_ACC_TRUNC</a></div><div class="ttdeci">#define H5F_ACC_TRUNC</div><div class="ttdef"><b>Definition</b> H5Fpublic.h:30</div></div>
<div class="ttc" id="a_h5_ipublic_8h_html_a0045db7ff9c22ad35db6ae91662e1943"><div class="ttname"><a href="_h5_ipublic_8h.html#a0045db7ff9c22ad35db6ae91662e1943">hid_t</a></div><div class="ttdeci">int64_t hid_t</div><div class="ttdef"><b>Definition</b> H5Ipublic.h:60</div></div>
<div class="ttc" id="a_h5_ppublic_8h_html_afa85e97bfbf9bf1c58e39263846c568f"><div class="ttname"><a href="_h5_ppublic_8h.html#afa85e97bfbf9bf1c58e39263846c568f">H5P_DEFAULT</a></div><div class="ttdeci">#define H5P_DEFAULT</div><div class="ttdef"><b>Definition</b> H5Ppublic.h:220</div></div>
<div class="ttc" id="a_h5_spublic_8h_html_a5f96eeee84b987f18470737f85af0484"><div class="ttname"><a href="_h5_spublic_8h.html#a5f96eeee84b987f18470737f85af0484">H5S_ALL</a></div><div class="ttdeci">#define H5S_ALL</div><div class="ttdef"><b>Definition</b> H5Spublic.h:33</div></div>
<div class="ttc" id="a_h5_zpublic_8h_html_afae8461c70d47e63be2163af23362237"><div class="ttname"><a href="_h5_zpublic_8h.html#afae8461c70d47e63be2163af23362237">H5Z_filter_t</a></div><div class="ttdeci">int H5Z_filter_t</div><div class="ttdoc">Filter identifiers.</div><div class="ttdef"><b>Definition</b> H5Zpublic.h:55</div></div>
<div class="ttc" id="a_h5public_8h_html_a3b079ecf932a5c599499cf7e298af160"><div class="ttname"><a href="_h5public_8h.html#a3b079ecf932a5c599499cf7e298af160">herr_t</a></div><div class="ttdeci">int herr_t</div><div class="ttdef"><b>Definition</b> H5public.h:269</div></div>
<div class="ttc" id="a_h5public_8h_html_a7f81cce70fb546af88da24d9285d3c1c"><div class="ttname"><a href="_h5public_8h.html#a7f81cce70fb546af88da24d9285d3c1c">hsize_t</a></div><div class="ttdeci">uint64_t hsize_t</div><div class="ttdef"><b>Definition</b> H5public.h:321</div></div>
<div class="ttc" id="a_h5public_8h_html_aa8f6c28736dbd0f18388c67911d38aca"><div class="ttname"><a href="_h5public_8h.html#aa8f6c28736dbd0f18388c67911d38aca">htri_t</a></div><div class="ttdeci">int htri_t</div><div class="ttdef"><b>Definition</b> H5public.h:299</div></div>
<div class="ttc" id="agroup___h5_d_html_ga98f44998b67587662af8b0d8a0a75906"><div class="ttname"><a href="group___h5_d.html#ga98f44998b67587662af8b0d8a0a75906">H5Dwrite</a></div><div class="ttdeci">herr_t H5Dwrite(hid_t dset_id, hid_t mem_type_id, hid_t mem_space_id, hid_t file_space_id, hid_t dxpl_id, const void *buf)</div><div class="ttdoc">Writes raw data from a buffer to a dataset.</div></div>
<div class="ttc" id="agroup___h5_d_html_gae47c3f38db49db127faf221624c30609"><div class="ttname"><a href="group___h5_d.html#gae47c3f38db49db127faf221624c30609">H5Dclose</a></div><div class="ttdeci">herr_t H5Dclose(hid_t dset_id)</div><div class="ttdoc">Closes the specified dataset.</div></div>
<div class="ttc" id="agroup___h5_f_html_gac55cd91d80822e4f8c2a7f04ea71b124"><div class="ttname"><a href="group___h5_f.html#gac55cd91d80822e4f8c2a7f04ea71b124">H5Fclose</a></div><div class="ttdeci">herr_t H5Fclose(hid_t file_id)</div><div class="ttdoc">Terminates access to an HDF5 file.</div></div>
<div class="ttc" id="agroup___h5_f_html_gae64b51ee9ac0781bc4ccc599d98387f4"><div class="ttname"><a href="group___h5_f.html#gae64b51ee9ac0781bc4ccc599d98387f4">H5Fcreate</a></div><div class="ttdeci">hid_t H5Fcreate(const char *filename, unsigned flags, hid_t fcpl_id, hid_t fapl_id)</div><div class="ttdoc">Creates an HDF5 file.</div></div>
<div class="ttc" id="agroup___h5_s_html_ga2b53128a39c8f104c1c9c2a91590fcc1"><div class="ttname"><a href="group___h5_s.html#ga2b53128a39c8f104c1c9c2a91590fcc1">H5Sclose</a></div><div class="ttdeci">herr_t H5Sclose(hid_t space_id)</div><div class="ttdoc">Releases and terminates access to a dataspace.</div></div>
<div class="ttc" id="agroup___h5_s_html_ga8e35eea5738b4805856eac7d595254ae"><div class="ttname"><a href="group___h5_s.html#ga8e35eea5738b4805856eac7d595254ae">H5Screate_simple</a></div><div class="ttdeci">hid_t H5Screate_simple(int rank, const hsize_t dims[], const hsize_t maxdims[])</div><div class="ttdoc">Creates a new simple dataspace and opens it for access.</div></div>
<div class="ttc" id="agroup___p_d_t_n_a_t_html_ga3cf93ffc6782be68070ef8e00f219ec2"><div class="ttname"><a href="group___p_d_t_n_a_t.html#ga3cf93ffc6782be68070ef8e00f219ec2">H5T_NATIVE_INT</a></div><div class="ttdeci">#define H5T_NATIVE_INT</div><div class="ttdef"><b>Definition</b> H5Tpublic.h:988</div></div>
<div class="ttc" id="agroup___p_l_c_r_html_ga5dce61149211d3ef319452aa598887fb"><div class="ttname"><a href="group___p_l_c_r.html#ga5dce61149211d3ef319452aa598887fb">H5Pclose</a></div><div class="ttdeci">herr_t H5Pclose(hid_t plist_id)</div><div class="ttdoc">Terminates access to a property list.</div></div>
</div><!-- fragment --><h1 class="doxsection"><a class="anchor" id="sec_compts_ineff"></a>
If a Compression Filter Was Not Effective</h1>
<p>There is no “one size fits all” compression filter solution. Users have to consider a number of characteristics such as the type of data, the desired compression ratio, the encoding/decoding speed, the general availability of a compression filter, and licensing among other issues before committing to a compression filter. This is especially true for data producers. The way data is written will affect how much bandwidth consumers will need to download data products, how much system memory and time will be required to read the data, and how many data products can be stored on the users’ system to name a few issues. Users should plan on experimenting with various compression filters and settings. The following are some suggestions for where to start finding the best compression filter for your data: </p><ul>
<li>Find the compression filter which operates best on the type of data in the file and for the objectives of the file users. Different applications, data providers, and data consumers will work differently with different compression filters. For example, <b>szip</b> compression is fast but typically achieves smaller compression ratios for floating point data than gzip. For more information on <b>szip</b>, see the <a class="el" href="_h5_d__u_g.html#subsubsec_dataset_filters_szip" title="Using the SZip Filter">Using the SZip Filter</a> page. </li>
<li>Once you have the right compression method, find the right parameters. For example, <b>gzip</b> compression at level 6 usually achieves a compression ratio comparable to level 9 in less time. For more information on compression levels, see the <a class="el" href="group___d_c_p_l.html#gaf1f569bfc54552bdb9317d2b63318a0d" title="Sets deflate (GNU gzip) compression method and compression level.">H5Pset_deflate</a> entry on the <a class="el" href="group___h5_p.html" title="Property Lists (H5P)">Property Lists (H5P)</a> page of the <a class="el" href="_r_m.html" title="HDF5 Reference Manual">HDF5 Reference Manual</a>. </li>
<li>Data preprocessing using a filter such as <b>shuffling</b> in combination with compression can drastically improve the compression ratio. See the <a class="el" href="#subsec_compts_ineff_comp" title="Compression Comparisons">Compression Comparisons</a> section below for more information.</li>
</ul>
<p>The <a class="el" href="_h5_t_o_o_l__r_p__u_g.html#sec_cltools_h5repack" title="h5repack">h5repack</a> tool can be used to experiment with the data to address the items above.</p>
<p>Users should also look beyond compression. An HDF5 file may contain a substantial amount of unused space. The <a class="el" href="_h5_t_o_o_l__s_t__u_g.html#sec_cltools_h5stat" title="h5stat">h5stat</a> tool can be used to determine if space is used efficiently in an HDF5 file, and the <a class="el" href="_h5_t_o_o_l__r_p__u_g.html#sec_cltools_h5repack" title="h5repack">h5repack</a> tool can be used to reduce the amount of unused space in an HDF5 file. See <a class="el" href="#subsec_compts_ineff_alt" title="An Alternative to Compression">An Alternative to Compression</a> for more information.</p>
<h2 class="doxsection"><a class="anchor" id="subsec_compts_ineff_comp"></a>
Compression Comparisons</h2>
<p>An extensive comparison of different compression filters is outside the scope of this document. However, it is easy to show that, unless a suitable compression method or an advantageous filter combination is chosen, applying the same compression filter to different types of data may not reduce HDF5 file size as much as possible.</p>
<p>For example, we looked at a NASA weather data product file packaged with it its geolocation information (the file name is <code>GCRIOREDRO_npp_d20030125_t0702533_e0711257_b00993_c20140501163427060570_XXXX_XXX.h5</code>) and used <a class="el" href="_h5_t_o_o_l__r_p__u_g.html#sec_cltools_h5repack" title="h5repack">h5repack</a> to apply three different compressions to the original file: </p><ul>
<li>gzip with compression level 7 </li>
<li>szip compression using NN mode and 32-bit block size </li>
<li>Shuffle in combination with gzip compression level 7</li>
</ul>
<p>Then we compared the sizes of the 32-bit floating dataset <code>/All_Data/CrIMSS-EDR-GEOTC_All/Height</code> when different types of compression were used and compared for the sizes of the 32-bit integer dataset <code>/All_Data/CrIMSS-EDR_All/FORnum</code>. The results are shown in the table below. </p><table class="doxtable">
<caption>Table 1: Compression ratio for different types of compressions when using h5repack</caption>
<tr>
<th>Data</th><th>Original</th><th>gzip Level 7</th><th>szip Using NN Mode and Blocksize 32</th><th>Shuffle and gzip Level 7</th></tr>
<tr>
<td>32-bit Floats</td><td>1</td><td>2.087</td><td>1.628</td><td>2.56</td></tr>
<tr>
<td>32-bit Integers</td><td>1</td><td>3.642</td><td>10.832</td><td>38.20</td></tr>
</table>
<p>The combination of the <b>shuffle</b> filter and <b>gzip</b> compression level 7 worked well on both floating point and integer datasets, as shown in the fifth column of the table above. <b>gzip</b> compression worked better than <b>szip</b> on the floating point dataset, but not on the integer dataset as shown by the results in columns three and four. Clearly, if the objective is to minimize the size of the file, datasets with different types of data have to be compressed with different compression methods.</p>
<p>For more information on the <b>shuffle</b> filter, see the <a class="el" href="_h5_d__u_g.html#subsubsec_dataset_transfer_filter" title="Data Pipeline Filters">Data Pipeline Filters</a> section in the <a class="el" href="_h5_d__u_g.html#sec_dataset" title="HDF5 Datasets">HDF5 Datasets</a> chapter of the <a class="el" href="_u_g.html" title="HDF5 User Guide">HDF5 User Guide</a>. See also the <a class="el" href="group___h5_p.html" title="Property Lists (H5P)">Property Lists (H5P)</a> in the <a class="el" href="_r_m.html" title="HDF5 Reference Manual">HDF5 Reference Manual</a> for the <a class="el" href="group___d_c_p_l.html#ga31e09cb0bf2da2893eed8a72220e6521" title="Sets up use of the shuffle filter.">H5Pset_shuffle</a> function call entry.</p>
<h2 class="doxsection"><a class="anchor" id="subsec_compts_ineff_alt"></a>
An Alternative to Compression</h2>
<p>Sometimes HDF5 files contain unused space. The <a class="el" href="_h5_t_o_o_l__r_p__u_g.html#sec_cltools_h5repack" title="h5repack">h5repack</a> command-line tool can be used to reduce the amount of unused space in a file without changing any storage parameters of the data. For example, running <a class="el" href="_h5_t_o_o_l__s_t__u_g.html#sec_cltools_h5stat" title="h5stat">h5stat</a> on the file <code>GCRIOREDRO_npp_d20030125_t0702533_e0711257_b00993_c20140501163524579819_XXXX_XXX.h5</code> shows: </p><div class="fragment"><div class="line">Summary of file space information:</div>
<div class="line"> File metadata: 425632 bytes</div>
<div class="line"> Raw data: 328202 bytes</div>
<div class="line"> Unaccounted space: 449322 bytes</div>
<div class="line">Total space: 1203156 bytes</div>
</div><!-- fragment --><p>After running <a class="el" href="_h5_t_o_o_l__r_p__u_g.html#sec_cltools_h5repack" title="h5repack">h5repack</a>, the file shows a 10-fold reduction in unaccounted space: </p><div class="fragment"><div class="line">Summary of file space information:</div>
<div class="line"> File metadata: 425176 bytes</div>
<div class="line"> Raw data: 328202 bytes</div>
<div class="line"> Unaccounted space: 45846 bytes</div>
<div class="line">Total space: 799224 bytes</div>
</div><!-- fragment --><p>There is also a small reduction in file metadata space. For more information on <a class="el" href="_h5_t_o_o_l__r_p__u_g.html#sec_cltools_h5repack" title="h5repack">h5repack</a> and <a class="el" href="_h5_t_o_o_l__s_t__u_g.html#sec_cltools_h5stat" title="h5stat">h5stat</a>, see the <a class="el" href="_u_g.html#sec_cltools" title="Command Line Tools for HDF5 Files">Command Line Tools for HDF5 Files</a> page in the <a class="el" href="_u_g.html" title="HDF5 User Guide">HDF5 User Guide</a>.</p>
<h1 class="doxsection"><a class="anchor" id="sec_compts_other"></a>
Other Resources</h1>
<p>See the following documents published by The HDF Group for more information. </p><ul>
<li>See the <a class="el" href="_l_b_com_dset.html#secLBComDsetCreate" title="Creating a Compressed Dataset">Creating a Compressed Dataset</a> tutorial. </li>
<li>The “Filter Behavior in HDF5” note is part of the <a class="el" href="group___o_c_p_l.html#gae7ac4c110e9ec3fcf38966928b96cda3" title="Adds a filter to the filter pipeline.">H5Pset_filter</a> function call entry. See the <a class="el" href="group___h5_p.html" title="Property Lists (H5P)">Property Lists (H5P)</a> in the <a class="el" href="_r_m.html" title="HDF5 Reference Manual">HDF5 Reference Manual</a>.</li>
</ul>
<hr />
<p> Navigate back: <a class="el" href="index.html" title="notitle">Main</a> / <a class="el" href="_t_n.html" title="Technical Notes">Technical Notes</a> </p>
</div></div><!-- contents -->
</div><!-- PageDoc -->
</div><!-- doc-content -->
<div id="page-nav" class="page-nav-panel">
<div id="page-nav-resize-handle"></div>
<div id="page-nav-tree">
<div id="page-nav-contents">
</div><!-- page-nav-contents -->
</div><!-- page-nav-tree -->
</div><!-- page-nav -->
</div><!-- container -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="footer">Generated by <a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.16.1 </li>
</ul>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink