mirror of
https://github.com/HDFGroup/hdf5.git
synced 2026-10-03 04:03:27 +03:00
Scot Breitenfeld
05676d1abe
* Consolidate documentation under doc/ directory
Move user-facing guides from release_docs/ and doxygen/ into a single
doc/ root. release_docs/ now holds only release artifacts (changelogs,
history, release process, maintainer info).
- git mv release_docs/INSTALL*.md, USING_*.md, README_HPC.md,
BuildSystemNotes.md, AutotoolsToCMakeOptions.md,
HDF5_Library_2.0.0_Migration_Guide.md → doc/
- git mv doxygen/ → doc/doxygen/
- Update CMakeLists.txt: HDF5_DOXYGEN_DIR and add_subdirectory path
- Update CMakeInstallation.cmake: all install paths for moved files
- Update bin/make_vers: hardcoded doxygen/ path substitution
- Update doc/doxygen/CMakeLists.txt: EXAMPLES_DIRECTORY and comments
- Update README.md, CONTRIBUTING.md, SECURITY.md, config/README.md,
release_docs/RELEASE_PROCESS.md: links to moved files
- Update doxygen .dox files: release_docs/ URLs for moved guides
- Rewrite release_docs/README.md for narrowed scope
* Add HDF5_DOCS_DIR variable for doc/ root path
Introduce HDF5_DOCS_DIR = \${HDF5_SOURCE_DIR}/doc so that
CMakeInstallation.cmake and future callers reference the doc/
directory symbolically rather than by hardcoded path.
HDF5_DOXYGEN_DIR is now derived from HDF5_DOCS_DIR.
152 lines
8.2 KiB
Plaintext
152 lines
8.2 KiB
Plaintext
|
|
/** \page UNICODE Using UTF-8 Encoding in HDF5 Applications
|
|
|
|
Navigate back: \ref index "Main" / \ref TN
|
|
<hr>
|
|
|
|
\section sec_unicode_intro Introduction
|
|
Text and character data are often discussed as though text means ASCII text. We even go so far as
|
|
to call a file containing only ASCII text a plain text file. This works reasonably well for English
|
|
(though better for American English than British English), but what if that plain text file is in
|
|
French, German, Chinese, or any of several hundred other languages? This document introduces the
|
|
use of UTF-8 encoding (see note 1), enabling the use of a much more extensive and flexible character
|
|
set that can faithfully represent any of those languages.
|
|
|
|
This document assumes a working familiarity with UTF-8 and Unicode. Any reader who is unfamiliar
|
|
with UTF-8 encoding should read the [Wikipedia UTF-8 article](https://en.wikipedia.org/wiki/UTF-8)
|
|
before proceeding; it provides an excellent primer.
|
|
|
|
For our context, the most important UTF-8 concepts are:
|
|
\li Multi-byte and variable-size character encodings
|
|
\li Limitations of the ASCII character set
|
|
\li Risks associated with the use of the term plain text
|
|
\li Representation of multiple language alphabets or characters in a single document
|
|
|
|
More specific technical details will only become important if they affect the specifics of
|
|
your application design or implementation.
|
|
|
|
\section sec_unicode_support How and Where Is UTF-8 Supported in HDF5?
|
|
HDF5 uses characters in object names (which are actually link names, but that's a story for a
|
|
different article), dataset raw data, attribute names, and attribute raw data. Though the
|
|
mechanisms differ, you can use either ASCII or UTF-8 character sets in all of these situations.
|
|
|
|
\subsection subsec_unicode_support_names Object and Attribute Names
|
|
By default, HDF5 creates object and attribute names with ASCII character encoding. An object or
|
|
attribute creation property list setting is required to create object names with UTF-8 characters.
|
|
This uses the function #H5Pset_char_encoding, which sets the character encoding used for object and attribute names.
|
|
|
|
For example, the following call sequence could be used to create a dataset with its name encoded with the UTF-8 character set:
|
|
|
|
\code
|
|
lcpl_id = H5Pcreate(H5P_LINK_CREATE) ;
|
|
error = H5Pset_char_encoding(lcpl_id, H5T_CSET_UTF8) ;
|
|
dataset_id = H5Dcreate2(group_id, "datos_ñ", datatype_id, dataspace_id,
|
|
lcpl_id, H5P_DEFAULT, H5P_DEFAULT) ;
|
|
\endcode
|
|
|
|
If the character encoding of an attribute name is unknown, the combination of an
|
|
#H5Aget_create_plist call and an #H5Pget_char_encoding call will reveal that information.
|
|
If the character encoding of an object name is unknown, the information can be accessed
|
|
through the object's H5L_info_t structure which can be obtained using #H5Lvisit or #H5Lget_info_by_idx calls.
|
|
|
|
\subsection subsec_unicode_support_char Character Datatypes in Datasets and Attributes
|
|
Like object names, HDF5 character data in datasets and attributes is encoded as ASCII by
|
|
default. Setting up attribute or dataset character data to be UTF-8-encoded is accomplished
|
|
while defining the attribute or dataset datatype. This makes use of the function #H5Tset_cset,
|
|
which sets the character encoding to be used in building a character datatype.
|
|
|
|
For example, the following commands could be used to create an 8-character, UTF-8 encoded,
|
|
string datatype for use in either an attribute or dataset:
|
|
|
|
\code
|
|
datatype_id = H5Tcopy(H5T_C_S1);
|
|
error = H5Tset_cset(datatype_id, H5T_CSET_UTF8);
|
|
error = H5Tset_size(datatype_id, "8");
|
|
\endcode
|
|
|
|
If a character or string datatype's character encoding is unknown, the combination of an
|
|
#H5Aget_type or #H5Dget_type call and an #H5Tget_cset call can be used to determine that.
|
|
|
|
\section sec_unicode_warn Caveats, Pitfalls, and Things to Watch For
|
|
Programmers who are accustomed to using ASCII text without accommodating other text
|
|
encodings will have to be aware of certain common issues as they begin using UTF-8 encodings.
|
|
|
|
\subsection subsec_unicode_warn_port Cross-platform Portability
|
|
Since the HDF5 Library handles datatypes directly, UTF-8 encoded text in dataset and
|
|
attribute datatypes in a well-designed HDF5 application and file should work transparently
|
|
across platforms. The same should be true of handling names of groups, datasets, committed
|
|
datatypes, and attributes within a file.
|
|
|
|
Be aware, however, of system or application limitations once data or other information
|
|
has been extracted from an HDF5 file. The application or system must be designed to
|
|
accommodate UTF-8 encodings if the information is then used elsewhere in the application or system environment.
|
|
|
|
Data from a UTF-8 encoded HDF5 datatype, in either a dataset or an attribute,
|
|
that has been established within an HDF5 application should "just work" within the HDF5 portions of the application.
|
|
|
|
\subsection subsec_unicode_warn_names Filenames
|
|
Since file access is a system issue, filenames do not fall within the scope
|
|
of HDF5's UTF-8 capabilities; filenames are encoded at the system level.
|
|
|
|
Linux and Mac OS systems normally handle UTF-8 encoded filenames correctly
|
|
while Windows systems generally do not.
|
|
|
|
\section sec_unicode_text The *Plain Text* Illusion
|
|
Beware the use of the term *plain text*. *Plain text* is at best ambiguous, but often
|
|
misleading. Many will assume that *plain text* means ASCII, but plain text German or
|
|
French, for example, cannot be represented in ASCII. Plain text is only unambiguous
|
|
in the context of English (and even then can be problematic!).
|
|
|
|
\subsection subsec_unicode_warn_store Storage Size
|
|
Programmers and data users accustomed to working strictly with ASCII data generally make
|
|
the reasonable assumption that 1 character, be it in an object name or in data, requires
|
|
1 byte of storage. This equation does not work when using UTF-8 or any other Unicode encoding.
|
|
With Unicode encoding, number of characters is not synonymous with number of bytes. One must
|
|
get used to thinking in terms of number of characters when talking about content, reserving
|
|
number of bytes for discussions of storage size.
|
|
|
|
When working with Unicode text, one can no longer assume a 1:1 correspondence between the
|
|
number of characters and the data storage requirement.
|
|
|
|
\subsection subsec_unicode_warn_sys System Dependencies
|
|
<b>Linux</b>, <b>Unix</b>, and similar systems generally handle UTF-8 encodings in correct and
|
|
predictable ways. There is an apparent consensus in the Linux community that "UTF-8 is just the right way to go."
|
|
|
|
<b>Mac OS</b> systems generally handle UTF-8 encodings correctly.
|
|
|
|
<b>Windows</b> systems use a different Unicode encoding, <a href="https://en.wikipedia.org/wiki/UTF-16">UCS-2</a>
|
|
(discussed in this UTF-16 article) at
|
|
the system level. Within an HDF5 file and application on a Windows system, UTF-8 encoding should
|
|
work correctly and as expected. Problems may arise, however, when that UTF-8 encoding is exposed
|
|
directly to the Windows system. For example:
|
|
\li File open and close calls on files with UTF-8 encoded names are likely to fail as the HDF5
|
|
open and close operations interact directly with the Windows file system interface.
|
|
\li Anytime an HDF5 command-line utility (\ref H5TOOL_LS_UG or \ref H5TOOL_DP_UG, for example) emits text output, the
|
|
Windows system must interpret the character encodings. If that output is UTF-8 encoded, Windows
|
|
will correctly interpret only those characters in the ASCII subset of UTF-8.
|
|
|
|
\section sec_unicode_common Common Characters in UTF-8 and ASCII
|
|
One interesting feature of UTF-8 and ASCII is that the ASCII character set is a discrete subset of
|
|
the UTF-8 character set. And where they overlap, the encodings are identical. This means that a
|
|
character string consisting entirely of members of the ASCII character set can be encoded in either
|
|
ASCII or UTF-8, the two encodings will be indistinguishable, and the encodings will require exactly the same storage space.
|
|
|
|
|
|
\section sec_unicode_also See Also
|
|
|
|
- For object and attribute names:
|
|
* #H5Pset_char_encoding
|
|
* #H5Pget_char_encoding
|
|
- For dataset and attribute datatypes:
|
|
* #H5Tset_cset
|
|
* #H5Tget_cset
|
|
- [UTF-8 article on Wikipedia](https://en.wikipedia.org/wiki/UTF-8)
|
|
|
|
<h3>NOTES</h3>
|
|
1. UTF-8 is the only Unicode standard encoding supported in HDF5.
|
|
|
|
<hr>
|
|
Navigate back: \ref index "Main" / \ref TN
|
|
|
|
*/
|