Harden selection decoding (#6525)

* Harden serialized selection decoding

* Clang-format and expand test

* Guard against rank 0 hyperslab selection in serialization

* Add CHANGELOG entry

* Modify CHANGELOG

---------

Co-authored-by: Alb3e3 <74142887+Alb3e3@users.noreply.github.com>

release_docs/CHANGELOG.md

@@ -203,6 +203,10 @@ The `h5repack` tool now obtains its default low and high library version bounds
 
    Fixed a bug where a flag in H5Cimage.c wasn't getting set correctly for release builds of HDF5, leading to incorrect error checking when reconstructing metadata cache entries.
 
+### Hardened decoding of serialized dataspace selections against malformed buffers
+
+   `H5S_select_deserialize()` and the per-selection-type deserialize callbacks (all, hyperslab, none, and point) previously computed the pointer to the last valid buffer byte as `buffer + size - 1` without first checking the buffer size. A buffer shorter than the 4-byte selection-type header, or a zero-length selection-info buffer, would underflow this computation and produce an out-of-bounds end pointer, defeating subsequent overflow checks. The deserialize routines now reject a buffer that is too small to hold the selection type, and they reject an empty selection-info buffer before deriving the end pointer. Hyperslab decoding additionally now rejects a serialized rank of 0 or greater than `H5S_MAX_RANK`. As a companion fix, `H5S__hyper_serialize()` now returns an error when asked to serialize a hyperslab selection on a rank-0 (scalar or null) dataspace, a state that can arise when a dataspace extent is collapsed to a scalar after a hyperslab selection has already been made.
+
 ## Java Library
 
 ## Configuration