mirrors/opencv
2
0
Fork 0
mirror of https://github.com/opencv/opencv.git synced 2026-09-25 04:09:57 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #29836 from velonica0:dnn-blocked-pointwise-span

Browse Source 
dnn: use the full vector width in blocked-layout pointwise kernels - #29836

Four block-layout kernels share one defect: they vectorize *across the channel block*, so they only use the vector unit while it happens to match `C0`. This PR fixes all four behind one shared helper, and along the way fixes an unrelated correctness bug found in the same function.

## 1. FP16 BatchNorm writes no output

`batch_norm2_layer.cpp`, the `CV_16F` arm of the accelerated block-layout path, wraps its whole body in a condition that cannot be true:

```cpp
} else if (type == CV_16F) {
    const hfloat* inptr = ...;
    if (type == CV_32F) {          // false by construction
```

There is no `else`, so every vector loop in that arm is dead and **nothing is written to the output**. A half-precision BatchNorm on a blocked tensor with `C0` equal to 1, 2 or 4 times the vector width returns whatever the destination buffer already held. The `CV_32F` and `CV_16BF` arms are correct; only the redundant wrapper is removed.

Confirmed with a sentinel-prefilled destination: before the fix the sentinel survives the call (`maxerr = 12346`) for all three `C0` values, after it the output is exact.

## 2. The kernels only use min(C0, VEC_SZ) lanes

`C0` is the block-layout channel block, fixed at 8 (`net_impl.hpp`, `DEFAULT_C0`), so these guards decide how much of the register gets used:

| kernel | guard | consequence with C0=8 |
|---|---|---|
| ChannelsPReLU | `C0 == VEC_SZ` | scalar at 4 lanes **and** at 16/32 |
| BatchNorm | `C0 == vlanes*{4,2,1}` | scalar at 16/32 |
| InstanceNorm | `c0 <= validC0 - VEC_SZ` | scalar at 16/32 |
| GroupNorm | `c0 <= c0_hi - VEC_SZ` | scalar at 16/32 |

PReLU is the worst case: an equality can only hold on an 8-lane build, so a default x86 SSE build (4 lanes) runs it scalar too. The other three use chunk loops that work at <= 8 lanes and fail above.

These files are not in the CPU-dispatch list, so `v_float32` is whatever `CPU_BASELINE` gives -- this is not RISC-V-specific.

### Approach

A block-layout plane is contiguous over `(H, W, C0)` and the coefficients repeat with period `C0`, so one register can span `vlanes/C0` pixels: replicate the coefficients across it and walk the plane flat.

All three spanning call sites now go through one helper, `cpu_kernels/blocked_pointwise.hpp::blockedSpanApply()`, parameterised by the per-element operation (`BlockedAffineOp`, `BlockedPReLUOp`). PReLU additionally gains a chunked path for `C0 > VEC_SZ`, which is what restores it on 4-lane targets. BatchNorm keeps its own unrolled loops and gains a whole-vector step plus a remainder loop. Pre-existing paths are untouched, so targets that already vectorized keep the same code.

**GroupNorm also gains a vector reduction.** Its mean/variance pass walked one channel at a time with a stride of `C0`, which no target vectorized at all, so that half speeds up everywhere rather than only on wide vectors. Both new GroupNorm paths are restricted to blocks owned entirely by one group; where a group boundary falls inside a block the old per-channel code still runs, because spanning would cross into channels another `parallel_for_` task is writing.

### Note for reviewers: the helper's `noinline` is load-bearing

`blockedSpanApply()` carries an explicit `noinline`. Inlined, GCC 15.2 on RISC-V speculates its stores into callers whose guard is false, which silently corrupted a neighbouring group's channels in `fastNormGroupBlockF32`. The symptom was exactly half the elements wrong at VLEN=1024 in the cases where a group splits a block; a runtime trace showed the guard evaluating false on every block, and inserting any call before the `if` made it disappear. Please do not remove the attribute.

## Benchmarks

SpacemiT K3, GCC 15.2, `CPU_BASELINE=RVV`, single thread, median of 11, pristine vs patched built back to back in one session. The board exposes two core types with different VLEN, so both columns are the same binary on the same machine.

Speedup, VLEN=256 / VLEN=1024:

| shape (NxC1xHxWxC0) | Ci | InstanceNorm | BatchNorm | GroupNorm |
|---|---|---|---|---|
| 1x32x56x56x8 | 256 | 1.01x / 14.38x | 0.91x / 11.62x | 1.85x / 14.56x |
| 1x16x28x28x8 | 128 | 1.01x / 16.27x | 0.98x / 3.95x | 1.91x / 16.01x |
| 1x8x112x112x8 | 64 | 1.02x / 12.76x | 1.03x / 1.37x | 1.98x / 12.86x |
| 1x16x56x56x4 | 64 | 3.01x / 12.47x | 4.49x / 6.18x | 3.01x / 11.78x |

ChannelsPReLU, measured separately the same way:

| shape | Ci | VLEN=256 | VLEN=1024 |
|---|---|---|---|
| 1x32x56x56x8 | 256 | 1.45x | 8.37x |
| 1x16x28x28x8 | 128 | 1.55x | 4.16x |
| 1x64x14x14x8 | 512 | 1.46x | 4.05x |
| 1x8x112x112x8 | 64 | 1.47x | 3.77x |
| 1x16x56x56x4 | 64 | 2.99x | 8.47x |
| 1x8x56x56x16 | 128 | 1.20x | 2.33x |

The VLEN=256 columns for InstanceNorm and BatchNorm are flat by construction -- `C0 == VEC_SZ` there, so those shapes already vectorized and the code is unchanged; the 0.91-1.03x spread is measurement noise. The rows that move at 256 are `C0=4` (block narrower than the vector), PReLU (broken at every width), and GroupNorm (reduction).

**Caveat on the BatchNorm numbers.** BatchNorm timings on this board are much less reproducible than the other three. The `1x8x112x112x8` case in particular measured anywhere from 1.4x to 6.9x across builds with byte-identical BatchNorm sources, and its pristine baseline moved by 26% between runs. The working set there is 3.06 MB, an exact multiple of 4096, and this hardware is sensitive to how source and destination alias in the cache; the numbers above are one back-to-back pair rather than a stable figure. InstanceNorm, GroupNorm and PReLU reproduced to within ~1% across every build.

## Testing

Verified against a scalar reference over 43 shape / `C0` / `Ci` / group combinations across the four layers, at VLEN 256 and 1024, at 1 and 8 threads, clean under `MALLOC_CHECK_=3`. Cases include partial trailing blocks, odd planes (7x7, 13x11) that exercise the remainder loops, `C0` from 2 to 64, and GroupNorm configurations where a group boundary falls inside a block -- those take the fallback and match bit-exactly, which is what confirms the ownership guard.

`opencv_test_dnn` was **not** run: the build used here is `BUILD_LIST=dnn`, which generates no dnn test target, and the board has no `opencv_extra` checkout. The blocked path's reachability was confirmed by inspection instead -- `ActivationLayer::getLayouts` passes the producer's layout through, and `useBlockLayout()` runs unconditionally in `finalizeGraph`, so these kernels are on the default path in real nets.

## Platform scope

Nothing here is behind a RISC-V `#ifdef`; this is universal-intrinsic code that compiles into every target.

| target (C0=8) | lanes | what changes |
|---|---|---|
| x86 SSE baseline (default) | 4 | PReLU newly vectorized; GroupNorm reduction newly vectorized |
| x86 AVX2 baseline | 8 | GroupNorm reduction; BatchNorm loop rewritten (same iterations) |
| x86 AVX-512 baseline | 16 | + all spanning paths go live |
| ARM64 NEON | 4 | as SSE baseline |
| ARMv7 NEON | 4 | `CV_SIMD_64F`=0, reduction path skipped |
| RVV 256 | 8 | PReLU, GroupNorm reduction |
| RVV >= 512 | 16/32 | everything |

Two changes reach a **default x86 build**: PReLU, which was scalar there because `C0 == VEC_SZ` cannot hold at 4 lanes, and the GroupNorm reduction, which was scalar everywhere. The latter changes GroupNorm's numerical output on those targets, since summation order differs -- measured at ~1e-7 relative here.

Measurements are RISC-V only; no x86 or ARM machine was available. The spanning branches were exercised on RVV at the same `vlanes/C0` ratios an AVX-512-baseline build would hit (2:1 and 4:1), but the x86-reachable changes above have had no x86 validation and are the part most worth checking in CI.
This commit is contained in:
velonica0
2026-09-06 11:02:41 +03:00
committed by GitHub
parent a427f0be44
commit ba7e716840
4 changed files with 338 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/dnn/src/layers/batch_norm2_layer.cpp
+50 -34
View File
@@ -146,7 +146,8 @@ static void batchnorm(const Mat& inp, Mat& out, const Mat& scale,
most part of the plane using vector code as if C0 == vlanes and
then process the tail using scalar code
*/
else if (C0 == vlanes*4 || C0 == vlanes*2 || C0 == vlanes) {
else if (C0 == vlanes*4 || C0 == vlanes*2 || C0 == vlanes ||
(C0 < vlanes && vlanes % C0 == 0)) {
// accelerated block layout case
int c = 0;
for (; c < c_delta; c++) {
@@ -156,6 +157,17 @@ static void batchnorm(const Mat& inp, Mat& out, const Mat& scale,
for (; c < MAX_UNROLL*vlanes; c++) {
scalebuf[c] = biasbuf[c] = 0.f;
}
// A vector wider than the channel block spans vlanes/C0 pixels,
// so repeat the C0-periodic coefficients across the register and
// step by a whole vector instead of by C0.
int cstep = C0;
if (C0 < vlanes) {
for (c = C0; c < vlanes; c++) {
scalebuf[c] = scalebuf[c - C0];
biasbuf[c] = biasbuf[c - C0];
}
cstep = vlanes;
}
v_float32 vsc0, vsc1, vsc2, vsc3;
v_float32 vb0, vb1, vb2, vb3;
vsc0 = vx_load(scalebuf);
@@ -194,47 +206,49 @@ static void batchnorm(const Mat& inp, Mat& out, const Mat& scale,
v_store(outptr + i + vlanes, x1);
}
} else {
for (; i < planesize_C0; i += C0) {
for (; i <= planesize_C0 - cstep; i += cstep) {
v_float32 x0 = vx_load(inptr + i);
x0 = v_fma(x0, vsc0, vb0);
v_store(outptr + i, x0);
}
for (; i < planesize_C0; i++)
outptr[i] = inptr[i]*scalebuf[i % C0] + biasbuf[i % C0];
}
} else if (type == CV_16F) {
const hfloat* inptr = (const hfloat*)inptr_;
hfloat* outptr = (hfloat*)outptr_;
if (type == CV_32F) {
if (C0 == vlanes*4) {
for (; i < planesize_C0; i += C0) {
v_float32 x0 = vx_load_expand(inptr + i);
v_float32 x1 = vx_load_expand(inptr + i + vlanes);
v_float32 x2 = vx_load_expand(inptr + i + vlanes*2);
v_float32 x3 = vx_load_expand(inptr + i + vlanes*3);
x0 = v_fma(x0, vsc0, vb0);
x1 = v_fma(x1, vsc1, vb1);
x2 = v_fma(x2, vsc2, vb2);
x3 = v_fma(x3, vsc3, vb3);
v_pack_store(outptr + i, x0);
v_pack_store(outptr + i + vlanes, x1);
v_pack_store(outptr + i + vlanes*2, x2);
v_pack_store(outptr + i + vlanes*3, x3);
}
} else if (C0 == vlanes*2) {
for (; i < planesize_C0; i += C0) {
v_float32 x0 = vx_load_expand(inptr + i);
v_float32 x1 = vx_load_expand(inptr + i + vlanes);
x0 = v_fma(x0, vsc0, vb0);
x1 = v_fma(x1, vsc1, vb1);
v_pack_store(outptr + i, x0);
v_pack_store(outptr + i + vlanes, x1);
}
} else {
for (; i < planesize_C0; i += C0) {
v_float32 x0 = vx_load_expand(inptr + i);
x0 = v_fma(x0, vsc0, vb0);
v_pack_store(outptr + i, x0);
}
if (C0 == vlanes*4) {
for (; i < planesize_C0; i += C0) {
v_float32 x0 = vx_load_expand(inptr + i);
v_float32 x1 = vx_load_expand(inptr + i + vlanes);
v_float32 x2 = vx_load_expand(inptr + i + vlanes*2);
v_float32 x3 = vx_load_expand(inptr + i + vlanes*3);
x0 = v_fma(x0, vsc0, vb0);
x1 = v_fma(x1, vsc1, vb1);
x2 = v_fma(x2, vsc2, vb2);
x3 = v_fma(x3, vsc3, vb3);
v_pack_store(outptr + i, x0);
v_pack_store(outptr + i + vlanes, x1);
v_pack_store(outptr + i + vlanes*2, x2);
v_pack_store(outptr + i + vlanes*3, x3);
}
} else if (C0 == vlanes*2) {
for (; i < planesize_C0; i += C0) {
v_float32 x0 = vx_load_expand(inptr + i);
v_float32 x1 = vx_load_expand(inptr + i + vlanes);
x0 = v_fma(x0, vsc0, vb0);
x1 = v_fma(x1, vsc1, vb1);
v_pack_store(outptr + i, x0);
v_pack_store(outptr + i + vlanes, x1);
}
} else {
for (; i <= planesize_C0 - cstep; i += cstep) {
v_float32 x0 = vx_load_expand(inptr + i);
x0 = v_fma(x0, vsc0, vb0);
v_pack_store(outptr + i, x0);
}
for (; i < planesize_C0; i++)
outptr[i] = hfloat(float(inptr[i])*scalebuf[i % C0] + biasbuf[i % C0]);
}
} else if (type == CV_16BF) {
const bfloat* inptr = (const bfloat*)inptr_;
@@ -264,11 +278,13 @@ static void batchnorm(const Mat& inp, Mat& out, const Mat& scale,
v_pack_store(outptr + i + vlanes, x1);
}
} else {
for (; i < planesize_C0; i += C0) {
for (; i <= planesize_C0 - cstep; i += cstep) {
v_float32 x0 = vx_load_expand(inptr + i);
x0 = v_fma(x0, vsc0, vb0);
v_pack_store(outptr + i, x0);
}
for (; i < planesize_C0; i++)
outptr[i] = bfloat(float(inptr[i])*scalebuf[i % C0] + biasbuf[i % C0]);
}
}
}
modules/dnn/src/layers/cpu_kernels/blocked_pointwise.hpp
+107
View File
@@ -0,0 +1,107 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
#ifndef OPENCV_DNN_BLOCKED_POINTWISE_HPP
#define OPENCV_DNN_BLOCKED_POINTWISE_HPP
#include <opencv2/core/hal/intrin.hpp>
namespace cv { namespace dnn {
#if (CV_SIMD || CV_SIMD_SCALABLE)
#if defined(__GNUC__)
#define CV_DNN_SPAN_NOINLINE __attribute__((noinline))
#elif defined(_MSC_VER)
#define CV_DNN_SPAN_NOINLINE __declspec(noinline)
#else
#define CV_DNN_SPAN_NOINLINE
#endif
/** True when the (H, W, C0) region of a block-layout tensor is a single contiguous
* run, which is what allows a block to be walked flat rather than row by row.
*/
static inline bool blockIsContiguous(int C0, int W,
size_t inStep2, size_t inStep3,
size_t outStep2, size_t outStep3)
{
return inStep3 == (size_t)C0 && inStep2 == (size_t)W * C0 &&
outStep3 == (size_t)C0 && outStep2 == (size_t)W * C0;
}
/** True when the block is contiguous and one register covers a whole number of
* pixels, i.e. when blockedSpanApply() may be used.
*/
static inline bool blockCanSpan(int C0, int VEC_SZ, int W,
size_t inStep2, size_t inStep3,
size_t outStep2, size_t outStep3)
{
return C0 < VEC_SZ && (VEC_SZ % C0) == 0 &&
blockIsContiguous(C0, W, inStep2, inStep3, outStep2, outStep3);
}
/** Applies a pointwise operation over one contiguous H*W*C0 block of a
* block-layout tensor, where the coefficients depend only on the channel and so
* repeat with period C0.
*
* Kernels that walk such a block channel-by-channel can only use min(C0, VEC_SZ)
* lanes of a register. Since the block is contiguous over (H, W, C0), replicating
* the coefficients across the register instead lets one iteration span VEC_SZ/C0
* pixels and keep every lane busy. Requires VEC_SZ % C0 == 0.
*
* Callers fill ca[0..C0) before the call; entries [C0, VEC_SZ) are filled here, so
* both arrays need room for VEC_SZ floats. Operations using a single coefficient
* pass the same array as ca and cb.
*
* Op supplies two overloads, one on v_float32 and one on float:
* operator()(x, a, b)
*
* Deliberately not inlined: GCC 15.2 on RISC-V speculates the stores below into
* callers whose guard is false, which corrupts elements the caller does not own
* (see fastNormGroupBlockF32, where a neighbouring group can share the block).
*/
template<typename Op>
CV_DNN_SPAN_NOINLINE
static void blockedSpanApply(const float* in, float* out, int64_t total,
int C0, int VEC_SZ, float* ca, float* cb, const Op& op)
{
for (int c = C0; c < VEC_SZ; ++c)
{
ca[c] = ca[c - C0];
cb[c] = cb[c - C0];
}
v_float32 va = vx_load(ca);
v_float32 vb = vx_load(cb);
int64_t idx = 0;
for (; idx <= total - VEC_SZ; idx += VEC_SZ)
vx_store(out + idx, op(vx_load(in + idx), va, vb));
for (; idx < total; ++idx)
{
int c = (int)(idx % C0);
out[idx] = op(in[idx], ca[c], cb[c]);
}
}
/** y = a*x + b -- BatchNorm, InstanceNorm and GroupNorm scale/shift. */
struct BlockedAffineOp
{
v_float32 operator()(const v_float32& x, const v_float32& a, const v_float32& b) const
{ return v_fma(x, a, b); }
float operator()(float x, float a, float b) const { return x * a + b; }
};
/** y = x >= 0 ? x : a*x -- per-channel PReLU. Ignores the second coefficient. */
struct BlockedPReLUOp
{
v_float32 operator()(const v_float32& x, const v_float32& a, const v_float32&) const
{ return v_select(v_ge(x, vx_setzero_f32()), x, v_mul(x, a)); }
float operator()(float x, float a, float) const { return x >= 0.f ? x : a * x; }
};
#endif // CV_SIMD || CV_SIMD_SCALABLE
}} // namespace cv::dnn
#endif // OPENCV_DNN_BLOCKED_POINTWISE_HPP
modules/dnn/src/layers/cpu_kernels/fast_norm.cpp
+142 -11
View File
@@ -4,6 +4,7 @@
#include "../../precomp.hpp"
#include "fast_norm.hpp"
#include "blocked_pointwise.hpp"
#include <opencv2/core/hal/intrin.hpp>
#include <type_traits>
@@ -359,6 +360,13 @@ static void fastNormChannelBlockF32(const Mat &input, const Mat &scale, const Ma
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int VEC_SZ = VTraits<v_float32>::vlanes();
// C0 is the block-layout channel block (8 by default), so on targets whose
// vector is wider than C0 the per-channel loops below never take a single
// iteration. One register spans VEC_SZ/C0 pixels there: replicate the
// C0-periodic coefficients across it and walk the contiguous H*W*C0 block
// flat, folding the partial per-channel sums at the end of the reduction.
const bool vecSpan = blockCanSpan(C0, VEC_SZ, W, inStep2, inStep3,
outStep2, outStep3);
#endif
// Accumulators are double-precision
@@ -371,9 +379,19 @@ static void fastNormChannelBlockF32(const Mat &input, const Mat &scale, const Ma
AutoBuffer<double> sumBuf(C0 * 2);
double* sum = sumBuf.data();
double* sqsum = sum + C0;
AutoBuffer<float> abBuf(C0 * 2);
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int abLen = std::max(C0, VEC_SZ);
#else
const int abLen = C0;
#endif
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
AutoBuffer<double> accBuf(VEC_SZ * 2);
double* accSum = accBuf.data();
double* accSqsum = accSum + VEC_SZ;
#endif
AutoBuffer<float> abBuf(abLen * 2);
float* alpha = abBuf.data();
float* beta = alpha + C0;
float* beta = alpha + abLen;
for (int i = r.start; i < r.end; ++i) {
int n = i / C1;
@@ -384,6 +402,67 @@ static void fastNormChannelBlockF32(const Mat &input, const Mat &scale, const Ma
const float* inbase = inptr0 + n * inStep0 + c1 * inStep1;
float* outbase = outptr0 + n * outStep0 + c1 * outStep1;
#if (CV_SIMD || CV_SIMD_SCALABLE)
if (vecSpan) {
const int64_t total = (int64_t)H * W * C0;
for (int c = 0; c < C0; ++c) {
sum[c] = 0.;
sqsum[c] = 0.;
}
int64_t idx = 0;
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
{
const int VEC_SZ_D = VTraits<v_float64>::vlanes();
const int reps = VEC_SZ / C0;
CV_DbgAssert(VEC_SZ == 2 * VEC_SZ_D);
v_float64 vsum_lo = vx_setzero_f64(), vsum_hi = vx_setzero_f64();
v_float64 vsqsum_lo = vx_setzero_f64(), vsqsum_hi = vx_setzero_f64();
for (; idx <= total - VEC_SZ; idx += VEC_SZ) {
v_float32 v = vx_load(inbase + idx);
v_float64 vlo = v_cvt_f64(v);
v_float64 vhi = v_cvt_f64_high(v);
vsum_lo = v_add(vsum_lo, vlo);
vsum_hi = v_add(vsum_hi, vhi);
vsqsum_lo = v_fma(vlo, vlo, vsqsum_lo);
vsqsum_hi = v_fma(vhi, vhi, vsqsum_hi);
}
vx_store(accSum, vsum_lo);
vx_store(accSum + VEC_SZ_D, vsum_hi);
vx_store(accSqsum, vsqsum_lo);
vx_store(accSqsum + VEC_SZ_D, vsqsum_hi);
// each register lane group holds a partial sum for one channel
for (int rp = 0; rp < reps; ++rp)
for (int c = 0; c < C0; ++c) {
sum[c] += accSum[rp * C0 + c];
sqsum[c] += accSqsum[rp * C0 + c];
}
}
#endif
for (; idx < total; ++idx) {
double v = (double)inbase[idx];
int c = (int)(idx % C0);
sum[c] += v;
sqsum[c] += v * v;
}
for (int c = 0; c < validC0; ++c) {
double mean = sum[c] * inv_norm_size_d;
double var = std::max(0., sqsum[c] * inv_norm_size_d - mean * mean);
float inv_stdev = 1.f / std::sqrt((float)var + epsilon);
alpha[c] = scale_data[cbase + c] * inv_stdev;
beta[c] = bias_data[cbase + c] - alpha[c] * (float)mean;
}
// zero coefficients turn the padding channels into zeros in the same pass
for (int c = validC0; c < C0; ++c) {
alpha[c] = 0.f;
beta[c] = 0.f;
}
blockedSpanApply(inbase, outbase, total, C0, VEC_SZ,
alpha, beta, BlockedAffineOp());
continue;
}
#endif
int c0 = 0;
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
const int VEC_SZ_D = VTraits<v_float64>::vlanes();
@@ -691,15 +770,27 @@ static void fastNormGroupBlockF32(const Mat &input, const Mat &scale, const Mat
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int VEC_SZ = VTraits<v_float32>::vlanes();
// A block-layout plane is contiguous over (H, W, C0). Whenever a whole block
// belongs to one group its H*W*C0 elements form a single contiguous run, which
// lets the reduction stream it instead of walking one channel at a time, and
// lets the affine pass span VEC_SZ/C0 pixels per register.
const bool blockContig = blockIsContiguous(C0, W, inStep2, inStep3,
outStep2, outStep3);
const bool vecSpan = blockContig && C0 < VEC_SZ && (VEC_SZ % C0) == 0;
#endif
parallel_for_(Range(0, N * (int)num_groups), [&](const Range& r) {
const float* inptr = (const float*)input.data;
float* outptr = (float*)output.data;
AutoBuffer<float> buf(C0 * 2);
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int abLen = std::max(C0, VEC_SZ);
#else
const int abLen = C0;
#endif
AutoBuffer<float> buf(abLen * 2);
float* alpha = buf.data();
float* beta = alpha + C0;
float* beta = alpha + abLen;
for (int i = r.start; i < r.end; ++i) {
int n = i / (int)num_groups;
@@ -708,17 +799,46 @@ static void fastNormGroupBlockF32(const Mat &input, const Mat &scale, const Mat
int c_end = c_start + channels_per_group;
double group_sum = 0., group_sqsum = 0.;
for (int c = c_start; c < c_end; c++) {
int c1 = c / C0;
int c0 = c % C0;
for (int c1 = c_start / C0, c1_last = (c_end - 1) / C0 + 1; c1 < c1_last; ++c1) {
const int cbase = c1 * C0;
const int g_lo = std::max(0, c_start - cbase);
const int g_hi = std::min(C0, c_end - cbase);
const float* inbase = inptr + n * inStep0 + c1 * inStep1;
for (int h = 0; h < H; ++h) {
const float* inrow = inbase + h * inStep2;
for (int w = 0; w < W; ++w) {
double v = (double)inrow[w * inStep3 + c0];
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
if (blockContig && g_lo == 0 && g_hi == C0) {
// every element of the block is summed, so order does not matter
const int64_t total = (int64_t)H * W * C0;
v_float64 vsum_lo = vx_setzero_f64(), vsum_hi = vx_setzero_f64();
v_float64 vsqsum_lo = vx_setzero_f64(), vsqsum_hi = vx_setzero_f64();
int64_t idx = 0;
for (; idx <= total - VEC_SZ; idx += VEC_SZ) {
v_float32 v = vx_load(inbase + idx);
v_float64 vlo = v_cvt_f64(v);
v_float64 vhi = v_cvt_f64_high(v);
vsum_lo = v_add(vsum_lo, vlo);
vsum_hi = v_add(vsum_hi, vhi);
vsqsum_lo = v_fma(vlo, vlo, vsqsum_lo);
vsqsum_hi = v_fma(vhi, vhi, vsqsum_hi);
}
group_sum += v_reduce_sum(vsum_lo) + v_reduce_sum(vsum_hi);
group_sqsum += v_reduce_sum(vsqsum_lo) + v_reduce_sum(vsqsum_hi);
for (; idx < total; ++idx) {
double v = (double)inbase[idx];
group_sum += v;
group_sqsum += v * v;
}
continue;
}
#endif
for (int c0 = g_lo; c0 < g_hi; ++c0) {
for (int h = 0; h < H; ++h) {
const float* inrow = inbase + h * inStep2;
for (int w = 0; w < W; ++w) {
double v = (double)inrow[w * inStep3 + c0];
group_sum += v;
group_sqsum += v * v;
}
}
}
}
@@ -741,6 +861,17 @@ static void fastNormGroupBlockF32(const Mat &input, const Mat &scale, const Mat
const float* inbase = inptr + n * inStep0 + c1 * inStep1;
float* outbase = outptr + n * outStep0 + c1 * outStep1;
#if (CV_SIMD || CV_SIMD_SCALABLE)
// Spanning is only safe when this group owns the whole block: the
// extra lanes of a wider vector would otherwise reach into channels
// that belong to a neighbouring group, which another task is writing.
if (vecSpan && c0_lo == 0 && c0_hi == C0) {
blockedSpanApply(inbase, outbase, (int64_t)H * W * C0,
C0, VEC_SZ, alpha, beta, BlockedAffineOp());
continue;
}
#endif
int c0 = c0_lo;
#if (CV_SIMD || CV_SIMD_SCALABLE)
for (; c0 <= c0_hi - VEC_SZ; c0 += VEC_SZ) {
modules/dnn/src/layers/elementwise_layers.cpp
+39 -2
View File
@@ -42,6 +42,7 @@
#include "../precomp.hpp"
#include "layers_common.hpp"
#include "cpu_kernels/blocked_pointwise.hpp"
#include "../op_cuda.hpp"
#include "../op_inf_engine.hpp"
#include "../ie_ngraph.hpp"
@@ -3949,14 +3950,27 @@ private:
const size_t outStep2 = dst.step.p[2] / sizeof(float);
const size_t outStep3 = dst.step.p[3] / sizeof(float);
#if CV_SIMD
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int VEC_SZ = VTraits<v_float32>::vlanes();
// C0 is the block-layout channel block (8 by default); VEC_SZ is whatever
// the target's float vector holds, so the two coincide only on 8-lane
// targets. Cover both directions instead: when C0 is a multiple of VEC_SZ
// each pixel is walked in VEC_SZ chunks, and when VEC_SZ is a multiple of
// C0 the slopes are replicated across the vector and the contiguous
// H*W*C0 block is walked in one flat loop.
const bool vecChunk = C0 > VEC_SZ && (C0 % VEC_SZ) == 0;
const bool vecFlat = blockCanSpan(C0, VEC_SZ, W, inStep2, inStep3,
outStep2, outStep3);
#endif
parallel_for_(Range(0, N * C1), [&](const Range& r) {
const float* inptr0 = src.ptr<float>();
float* outptr0 = dst.ptr<float>();
#if (CV_SIMD || CV_SIMD_SCALABLE)
AutoBuffer<float> slopeBuf(std::max(C0, VEC_SZ));
#else
AutoBuffer<float> slopeBuf(C0);
#endif
float* slopes = slopeBuf.data();
for (int i = r.start; i < r.end; ++i) {
@@ -3973,7 +3987,7 @@ private:
const float* inbase = inptr0 + n * inStep0 + c1 * inStep1;
float* outbase = outptr0 + n * outStep0 + c1 * outStep1;
#if CV_SIMD
#if (CV_SIMD || CV_SIMD_SCALABLE)
if (C0 == VEC_SZ) {
v_float32 vslope = vx_load(slopes);
v_float32 vzero = vx_setzero_f32();
@@ -3989,6 +4003,29 @@ private:
}
continue;
}
if (vecChunk) {
v_float32 vzero = vx_setzero_f32();
for (int h = 0; h < H; ++h) {
const float* inrow = inbase + h * inStep2;
float* outrow = outbase + h * outStep2;
for (int w = 0; w < W; ++w) {
const float* in_pos = inrow + w * inStep3;
float* out_pos = outrow + w * outStep3;
for (int c0 = 0; c0 < C0; c0 += VEC_SZ) {
v_float32 v = vx_load(in_pos + c0);
v_float32 scaled = v_mul(v, vx_load(slopes + c0));
v_float32 out = v_select(v_ge(v, vzero), v, scaled);
vx_store(out_pos + c0, out);
}
}
}
continue;
}
if (vecFlat) {
blockedSpanApply(inbase, outbase, (int64_t)H * W * C0, C0, VEC_SZ,
slopes, slopes, BlockedPReLUOp());
continue;
}
#endif
for (int h = 0; h < H; ++h) {
const float* inrow = inbase + h * inStep2;