mirrors/opencv
2
0
Fork 0
mirror of https://github.com/opencv/opencv.git synced 2026-09-25 04:09:57 +03:00
Code Issues Packages Projects Releases Wiki Activity

Broadcasting element-wise engine for cv::Mat (+ cv::texpr) (#29426)

Browse Source 
* experimental new arithmetics; work-in-progress

* continue working on new-gen arithmetic expressions

* * improved performance of the new add on small arrays
* added sub
* extended tests

* fixed potential bug when adding multi-channel array and a single-channel scalar

* improved const handling

* * added copyMask
* added mul/dev (without scale so far)

* * accelerated mul
* addedd scale to mul and div

* * done substantial refactoring; however a few more rounds of refactoring are ahead.
* added min, max, absdiff, addweighted.

* improved performance of the new arithmetic functions, but some of them are still slow, e.g. operations with mask have some bugs (that affect speed, not accuracy).

* * further (significantly) accelerated several functions, especially on small arrays: mul, binary ops with mask

* further polished the new arithmetic engine

* started integration of the new element-wise arithmetic engine into core

* big step forward. We now use the new engine inside cv::add, subtract, multiply, divide, absdiff, min and max.

* big progress:
* added bitwise operations
* fixed and accelerated compare
* ported regression tests to test new broadcasting behaviour of arithmetic functions

* lot's of improvements in compare, divide, addWeighted!

* lot's of small and big performance improvements in the new arithmetics

* * some more optimizations; parsing texpr-expressions is now faster as well

* port new_arithm to Linux/x86: dispatch guards, scalar-Mat compat fallback, dnn shape-contract fixes

Core:
- arithm.simd.hpp: CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY guards (the file is included
  once per dispatched mode on x86), vx_load_expand instead of the 128-bit v_load_expand,
  VTraits::vlanes() instead of ::nlanes
- arithm.cpp/precomp.hpp: compat fallback for scalar-like Mat operands (1x1, 1xcn/cnx1,
  4x1 CV_64F - java/python tuples, operator-(Mat, Matx)): treated as a per-channel scalar
  ONLY when the shapes are not broadcast-compatible, so every valid numpy-style broadcast
  keeps its meaning and calls that would otherwise throw get the 4.x semantics

DNN (fallout of the stricter shape semantics, found by the new engine):
- dict.hpp: DictValue relied on fresh AutoBuffer having size()==fixed_size; allocate explicitly
- batch_norm: weights_/bias_ are 1-D [n] now; 0/1-D forward runs on exact-shape 1-D views
- net_impl2: extend the post-forward sanity check to non-temp outputs - a layer that
  reallocates its preallocated output tensor now fails loudly instead of silently
  detaching the result from the graph
- LSTM/LSTM2 batchwise (layout=1): getMemoryShapes now matches what forward() writes
  (ONNX: Y=(batch,seq,dirs,hid), Yh/Yc=(batch,dirs,hid)); forward assembles seq-major
  results in a local buffer and transposes INTO the preallocated outputs in place

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* u8/s8 multiply: exact integer SIMD path on non-FP16 builds (3-10x vs 5.x)

The unit-scale branch of vecBinaryKernel already supported a separate work-vector
type Wvec1 (used by the ARM f16 build and by u16/s16 everywhere), but on x86 the
u8/s8 same-type multiply still went through the f32 hub. Route it through
v_uint16/v_int16: products of 8-bit values fit exactly (255^2 < 2^16), the
saturating pack on store gives bit-exact results at half the vector traffic.

Also fix a latent kernel bug this exposed: the unit-scale branch stepped by
Wvec's lane count while loading/storing Wvec1 vectors. All previous Wvec1
instantiations had equal lane counts, but u8's v_uint16 has 2x the lanes of
v_float32 - the pairs overlapped (50% redundant work) and the tail backoff
could write VECSZ bytes past the row end. The branch now derives its step,
offsets and tail condition from Wvec1 itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* vecBinaryKernel: constexpr Op::useScalar instead of a runtime-only scale check

Every binary op functor now declares whether it consumes the scale scalar
(params[0]): true only for mul and the two div variants. Ops that ignore it
(add/sub/min/max/absdiff) take the fast 2-arg branch unconditionally - the
'scalar == 1' check used to fail for them (their params[0] is 0), sending them
through the preproc branch, and the condition now folds at compile time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* restore cv::hal::mul8u as a wrapper over the element-wise engine

The symbol is still declared in core/hal/hal.hpp and called directly by external
code (the G-API fluid backend in opencv_contrib), but its implementation went
away with the old arithm kernels. Forward it to getMulFunc(CV_8U, CV_8U) - with
scale==1 it lands on the new exact integer SIMD path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* silence every new-arithm warning reported by CI (ARM64/Mac) and gcc 15

- arithm.cpp: bitwise_op_ocl and the actualScalarDepth/coerceTypes helpers are
  consumed only by the OpenCL paths - guard them with HAVE_OPENCL; haveScalar in
  cv::compare is read only inside CV_OCL_RUN - CV_UNUSED for OpenCL-less builds
- arithm_expr.hpp: declare getBitwiseFunc/getNotFunc/getAddWeightedFunc next to
  the other per-op entry points (-Wmissing-prototypes in arithm.dispatch.cpp)
- arithm.simd.hpp: define CV_SIMD_16F to 0 when FP16 SIMD is absent (-Wundef);
  {}-init the expandScalar staging buffers (-Wmaybe-uninitialized: they are
  fully written before use, but the compiler cannot prove it with runtime
  vector widths); rename the compare kernel's lambda parameter (-Wshadow)
- arithm_expr.cpp: rename the exec tile-lambda's hot-field locals that shadowed
  TExpr members and outer locals (-Wshadow)
- test_new_arithm_extensive.cpp: rename the name-generator lambdas' parameter
  shadowing the INSTANTIATE macro's own (-Wshadow), drop an unused variable

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* restore 4.x scalar semantics for the bindings' 4x1 CV_64F Scalar columns

The python/java bindings materialize numbers and tuples as a 2-D 4x1 CV_64F
Mat - or UMat, when the call carries UMat arguments. Three CI-reported python
failures came from those pseudo-scalars reaching the engine as arrays:

- absdiff(int_arr, 0): the (4,1) column is broadcast-COMPATIBLE with a 1-D
  array, so numpy semantics silently won - an outer-product f64 result instead
  of the int per-channel-scalar one;
- subtract(u8 4x8x4, (40,)): same, by the rows==4 coincidence;
- multiply(UMat, 2., dst=UMat): the scalar arrives as a UMAT, which the
  scalar detection did not recognize at all.

isScalarArg now treats the exact bindings shape - 2-D 4x1 CV_64F single-channel
Mat/UMat against a <=4-channel array - as a scalar UNCONDITIONALLY (a 1-D [4]
array has dims==1 and still broadcasts). One exception, decided in arithm_op:
when the partner is itself a tiny scalar-shaped array, both are honest data and
ride the broadcast (compare(Mat 4x1, Mat 1x1) - issue #8999 - stays elementwise).

Small-array discipline, this all runs per engine call: the probes read Mat/UMat
fields directly (rows == 4 alone rejects almost everything, no _InputArray
getter dispatch), and a UMAT scalar's 32 bytes are copied into a caller-stack
buffer - no heap, no getMat mapping. Measured: no latency change on 4x4/16x16
element-wise calls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* cv::texpr: std::string_view -> const std::string& in the public API

string_view in an exported signature breaks some CUDA toolchain builds, and for
short expression strings the difference is immaterial (SSO, parsed once). The
parser internals keep string_view - the argument converts implicitly. Also drop
the now-unused <string_view> include from cvstd.hpp, so the header does not
reach every nvcc TU.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* compare boundary-rewrite: fixed 4-slot kind/bound arrays instead of AutoBuffers

A CONST operand is capped at 4 channels (addConst), so the per-channel
kind/bound staging needs no dynamic buffers - plain int[4]/double[4], with a
CV_Assert on the contract. This is also what gcc's -Wmaybe-uninitialized was
flagging (it could not see the AutoBuffer's inline storage get filled).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* element-wise engine: unary math kernels (sqrt/exp/log/sin/cos/tanh/erf/relu) + select

New dispatched pair math.simd.hpp / math.dispatch.cpp - the unary/ternary sibling
of arithm.simd.hpp:

- vecUnaryKernel: T -> T over f16/bf16/f32/f64 on top of the intrin_math
  primitives (v_exp/v_log/v_sin/v_cos/v_sqrt/v_erf/v_max). f32/f64 compute
  natively, f16/bf16 ride the f32 hub inside the kernel (vx_load_pair_as /
  v_store_pair_as) - no materialized casts. Continuity collapse + the halide
  right-edge backoff, suppressed in-place (it would re-apply Op to
  already-written values). tanh = (e^2x-1)/(e^2x+1) with the input clamped to
  +/-10 (f32) / +/-20 (f64) - unclamped saturation hits inf/inf = NaN. erf has
  no f64 SIMD primitive: std::erf per lane.
- selectKernel(mask, x, y): 1-byte mask expanded to lane width and tested
  against zero in the INTEGER domain (immune to DAZ/FTZ), branches of any
  depth by element size, broadcast branches supported.
- emitUnary: math over a float input is T -> T now (f16 in -> f16 out, native
  kernel when input and result depths match); integer inputs still compute in
  the float domain and land in f32.
- emitTernary/select: literal branches are typed via typedConstFrom (an
  OP_CAST of a depth-less flex const crashed); a non-1-byte mask is normalized
  by an explicit "mask != 0" compare, never a value cast.

texpr already parsed the function names - they now execute. Tests: per-depth
accuracy of all 8 ops against the double std:: reference, integer input,
in-place, select over 4 depths / const branch / float mask.

Perf vs the classic kernels (1920x1080 f32, 16 threads, AVX2): exp 3.7x,
log 2.6x, sqrt 5.7x faster; polarToCart expressed as (r*cos(a), r*sin(a)) 4.0x.
Accuracy improves too (max rel err vs f64 reference): exp 8.1e-8 vs 2.1e-7,
log 8.0e-8 vs 1.5e-7.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* engine: OP_COPY_MASK folded into OP_SELECT; selectKernel moved to arithm.simd.hpp

copyMask(dst, mask, src) is select(mask, src, dst) - one masking primitive
instead of two. The compiler emits the masked-op tail as
addInsn(OP_SELECT, mask, r, out, out): the output slot rides as both arg2 and
the result, so unmasked elements are preserved by reading them back through
the b-branch. OP_COPY_MASK, copyMaskKernel and getCopyMaskFunc are gone.

selectKernel (moved from math.simd.hpp to arithm.simd.hpp) inherits every
copyMaskKernel optimization:
- the interleaved multichannel fast path (2..4 channels under a per-pixel
  mask: expand the mask once per VECSZ rows, v_store_interleave across lanes);
- the per-row scalar path with the row-skip when the selected source row IS
  dst (the "leave the output untouched" half of copyMask);
- plus the select-specific ones: branch broadcasts (stepx == 0) and the
  right-edge tail backoff under dst-aliases-a-branch - safe because re-running
  select over already-blended elements is idempotent; only dst == mask keeps
  the backoff off (the store would rewrite mask bytes before the re-read).

Masked-add perf is on par with the old copyMask (1280x720, 1 thread: 8UC3
204 -> 198 us, 32FC3 1395 -> 1344, 8UC1/32FC1 within noise). Full core suite
24117 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* engine: dedicated vectorized pow kernel (moved from arithm to math.simd.hpp)

pow was the last scalar-only binary op (scalarBinaryKernel + std::pow, f32/f64
only). The new powKernel keeps exact std::pow semantics and is T x T -> T over
all four float depths (f16/bf16 via the f32 hub):

- scalar exponent (the dominant call shape - texpr literals ride as 0-dim
  broadcast consts) is dispatched PER ROW to the special cases:
  y==2 -> x*x, y==3 -> x*x*x, y==0.5 -> v_sqrt, y==1 -> copy, y==0 -> fill 1;
- everything else - including a per-element exponent array - runs the general
  vectorized exp(y * log(x)) path, valid for x > 0; a vector pair containing
  any x <= 0 lane falls back to scalar std::pow for that pair (v_check_any),
  which preserves every std::pow subtlety: signed results for integer y on
  negative bases, NaN for fractional y, the x == 0 family;
- no right-edge tail backoff: pow is not idempotent, in-place calls finish
  rows in the scalar tail.

Perf vs the classic cv::pow (1920x1080 f32, 16 threads, AVX2): p=2 1.4x
(classic special-cases it too), p=3 6.1x, p=0.5 6.3x, fractional p 4.5x with
slightly better accuracy (6.1e-7 vs 7.4e-7 max rel err). Tests: exponent
sweep 2/3/0.5/1/0/2.5/-1.5 vs the double std::pow reference on f32/f64,
negative bases (exact signed cubes, NaN for fractional), array exponent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* powKernel: halide right-edge tail backoff in every SIMD loop

Same shape as vecBinaryKernel: the final partial vector re-processes
[width - VECSZ*2, width) instead of finishing scalar, suppressed when dst
aliases an input (pow is not idempotent - the overlap region must be
recomputed from an untouched source, which the no-alias case guarantees).

Modest measured win (~2% on ROI rows for the special-cased exponents; the
general path tail was already cheap - modern libm powf is fast), no
regressions; mainly aligns the kernel with the house style, where every
SIMD loop ends vector-wide.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix MSVC 2019 C2975: function-local constexpr as a template argument inside a lambda

MSVC 2019 loses the constexpr-ness of function-local constants (LOCAL_OPS,
MAX_DIMS, ...) when they are used as template arguments inside a lambda body
(AutoBuffer<Slice, LOCAL_OPS> / std::array<int, MAX_DIMS> in the parallel
bodies of BroadcastOp::run and TExpr::exec). Hoist them to namespace scope -
no behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* ocl_arithm_op: route 16U multiply to the CPU engine on Apple OpenCL

The Apple OpenCL driver miscompiles the 16U multiply kernel: products near
the top of the u16 range come back wrapped instead of saturated (CPU vs GPU
NORM_INF up to 65535 in OCL_Arithm/Mul.Mat CV_16U cases). The same arithm.cl
kernel is correct on Intel NEO and NVIDIA drivers - verified not to reproduce
on Linux/Intel iGPU - so gate the decline to __APPLE__ only; the CPU engine
computes 16u multiply exactly (integer SIMD path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* engine: neg/abs as compositions, clamp kernel, ** and ?: operators, abs(a-b) peephole

- OP_NEG and OP_ABS need no kernels: neg = sub(0, a), abs = absdiff(a, 0) -
  including the engine absdiff auto-type rule (signed |a| lands in the
  UNSIGNED type of the same width: |SHRT_MIN| fits u16 exactly instead of
  saturating; NB the public cv::absdiff auto depth keeps the source type for
  4.x compatibility - values agree, the depth rule is the engine own).
- peephole: abs(x - y) rewrites to absdiff(x, y) ALWAYS. On integers the
  literal semantics differ (the subtract saturates first: u8 gives
  max(x-y, 0)), but whoever writes abs(a - b) means absdiff - we deliberately
  hand out the useful semantics instead of the saturation artifact. The just-
  emitted OP_SUB is retired via the moveToOutput manoeuvre, so the program
  shrinks to the single absdiff instruction. abs(x), abs(x - 0) and
  absdiff(x, 0) all give one result.
- OP_CLAMP kernel (arithm.simd.hpp): v_min(v_max(x, lo), hi) over
  u8/s8/u16/s16/u32/s32/f32 (+f64 with 64-bit SIMD), scalar f16/bf16/64-bit
  ints; lo/hi may broadcast (the common clamp(img, a, b) shape) or be full
  arrays; the tail backoff stays on under dst-aliases-x (clamp is idempotent).
  emitTernary types literal bounds via typedConstFrom (same flex-const crash
  select had) and keeps the auto result type pinned to x.
- parser: "a ** b" == pow(a, b), precedence above * /, RIGHT-associative
  (a ** 2 ** 3 == a ** 8); "cond ? a : b" == select(cond, a, b), precedence
  below everything, right-associative chains (f1 ? a : f2 ? b : c) work
  without parentheses. parseTernary() is the expression entry point now.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* cv::exp/log/sqrt on the engine via math_op; hal functions wrapped as engine kernels

math_op is the master function of the unary math family (the arithm_op
analogue): same-shape same-type output over f16/bf16/f32/f64 (classic
exp/log accepted f32/f64 only - the half floats are new), two tiers:
- small (<= 100000 elements) and continuous: call the kernel DIRECTLY over
  the flattened data - no TExpr, no broadcastOp, no parallel_for setup;
- everything else: the usual single-instruction program via compile()/exec()
  (parallelism for large arrays, real steps for ROIs).

getMathFunc routes OP_EXP/OP_LOG at f32/f64 through the full cv::hal stack -
an external vendor HAL (CALL_HAL), IPP, or the built-in table kernels,
whichever is installed - by wrapping hal::exp32f/exp64f/log32f/log64f as
engine kernels with the function pointer in TKernel::userdata, the same
mechanism castKernel uses for core BinaryFuncs. The engine adds tiling and
parallelism on top, so every tier gets the best available scalar-span
implementation. v_exp/v_log remain for f16/bf16 (the f32 hub) and the ops
hal has no entry points for.

v_log_default_32f: the degree-8 polynomial is evaluated by Estrin pairing
(4 dependent levels) instead of an 8-FMA Horner chain (~5% on the f16 hub
path). An exp64 Taylor-without-division rewrite was tried and benched SLOWER
than the Cephes Pade scheme (the evaluation is FMA-throughput-bound, and
vdivpd pipelines well enough) - reverted; a table-based reduction is the only
way further there.

cv::exp f32 (16 threads, AVX2+IPP build), old -> new: 640 elements
0.16 -> 0.15 us, 16k 2.79 -> 2.49, 640x480 47 -> 20, 1920x1080 452 -> 60 us
(the old CPU loops were single-threaded); f64 exp 1080p 1295 -> 161 us.
No size regresses; small arrays now run at installed-HAL speed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* getMathFunc: IPP tier + raw-HAL probing; the exp/log table kernels are deleted

The raw cv_hal_* entry points return int for a reason: without an installed
HAL they are stubs returning CV_HAL_ERROR_NOT_IMPLEMENTED. getMathFunc now
selects the exp/log implementation in three tiers:
 1. HAVE_IPP && ipp::useIPP(): ippsExp/ippsLn through thin int adapters (IPP
    is not routed through the cv_hal_ hooks, so it needs its own tier);
 2. the raw cv_hal_exp32f/... hook, PROBED once with a 1-element call on the
    safe input 1.0 (cached in magic statics): implemented -> wrapped as an
    engine kernel with the function pointer in TKernel::userdata;
 3. the engine own v_exp/v_log kernels.
Whichever wins, the engine adds tiling and parallelism on top.

The EXPTAB/LOGTAB table kernels and their tables (~790 lines in
mathfuncs_core.simd.hpp + mathfuncs.cpp) are DELETED: they benched within
~15% of v_exp/v_log, not worth a second implementation. The public
cv::hal::exp32f/exp64f/log32f/log64f keep their contract - CALL_HAL, then
IPP, then the built-in implementation - but the built-in is now the engine
vector kernel via ew::mathSpanEngine (one contiguous span, exported from
math.dispatch.cpp).

All unary math kernels (vec/scalar/hal wrappers, pow) also handle the
vertical-broadcast tile (s0y == 0, a row expanded into a matrix): the first
row is computed, the rest are memcpy of it - transcendentals cost far more
than a row copy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* texpr: hypot(x, y) binary op (alias: mag)

hypot = sqrt(x^2 + y^2), NAIVE like cv::magnitude (not the overflow-safe
std::hypot), computed in the float work type; kernels for the four float
depths only (T x T -> T; integer inputs ride the usual f32-compute + cast).
A 10-line EwHypot functor on top of vecBinaryKernel in arithm.simd.hpp -
broadcast branches, continuity collapse and the tail backoff come for free.
Registered in the parser as both "hypot" (the C/numpy name) and "mag" (the
cv::magnitude-flavored alias). A building block for the future cartToPolar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* texpr: atan2(y, x) binary op - radians, standard C range

v_atan2 (arithm.simd.hpp, generic over the universal-intrinsic float vector):
the fastAtan2 minimax polynomial from mathfuncs_core v_atan_f32 reworked to
plain radians - the 180/pi factor dropped from the coefficients and the C
quadrant logic instead of the [0, 360) wrap, so the result matches std::atan2
over (-pi, pi]. Measured absolute accuracy ~1.6e-4 rad. (v_atan_f32 itself is
untouched - cv::phase/fastAtan2 keep their degree semantics.)

EwAtan2 rides vecBinaryKernel: f16/bf16/f32 through v_atan2 (the f32 hub),
f64 through exact scalar std::atan2. arg0 = y, arg1 = x, like std::atan2;
float depths only, same emitBinary policy as pow/hypot. Parser name "atan2".
Together with hypot this completes the cartToPolar building blocks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix the RISC-V RVV build and two ARM64 warnings

The new f64 kernel registrations (hypot, pow, the unary math family) gate on
CV_SIMD_64F || CV_SIMD_SCALABLE_64F, but the vx_setall_as(const double*,
v_float64&) helper family in arithm.simd.hpp was still CV_SIMD_64F-only -
scalable platforms (RVV) have v_float64 with CV_SIMD_64F == 0, so
vecBinaryKernel<double, ...> failed to instantiate there. Widen the helper
gate to match (verified with a riscv64 rv64gcv cross-build of opencv_core -
the engine f64 paths now vectorize on RVV instead of not compiling).

cv::exp/cv::log: the depth local is consumed by CV_OCL_RUN only - CV_UNUSED
for OpenCL-less builds (ARM64 -Wunused-variable).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* silence the remaining ARM64 gcc warnings

- compare boundary-rewrite: {}-init the fixed kind/bound arrays (filled for
  every channel used below, but gcc cannot prove it across the cn <= 4 loop);
- cv::exp/log: [[maybe_unused]] on the depth local (consumed by CV_OCL_RUN
  only), instead of the CV_UNUSED idiom;
- AutoBuffer::reserve: a targeted -Wmaybe-uninitialized suppression around
  the live-element copy loop - only [0, sz) is read, all written before, but
  gcc inlining a grow-from-inline-storage chain cannot see that. An
  annotation for the analyzer, no behavior change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* saturating 32-bit add/sub kernels; cv::texpr python binding; two CI warnings

- v_add_sat/v_sub_sat for v_int32/v_uint32, local to arithm.simd.hpp for now
  (the plan is to grow them into proper universal intrinsics later): NEON
  single-instruction vqadd/vqsub, elsewhere the Hacker Delight bit tricks
  over universal intrinsics (u32 add is 2 ops: or with the wrapped-compare
  mask). EwAdd/EwSub overload vec() for the 32-bit lanes and getAddSubFunc
  routes 32S/32U T->T through vecBinaryKernel instead of the former pure
  scalar kernel. Semantics unchanged - the scalar int64 tail already
  saturated; directed boundary tests added (both rails, 0 - INT_MIN, u32
  cases, a full-range random block vs an exact int64 reference).
  640x480 32S add: 0.48x of 5.x -> parity (memory-bound); 1080p: 6-8x.

- cv::texpr becomes CV_EXPORTS_W: python gets cv.texpr(expr, [inputs]) ->
  tuple of ndarrays, so `res, = cv.texpr(...)` and `mag, ang = cv.texpr(...)`
  unpacking both work. modules/python/test/test_expr.py covers arithmetic,
  the fused abs(a-b), casts, broadcasting, ?: and ** operators, math
  functions vs numpy, clamp, named temporaries, tuple outputs, the one-line
  cartToPolar and the int32 saturation cases.

- warnings: {}-init the parser args array (gcc -Wmaybe-uninitialized on
  Ubuntu 20/22); the compare short-row block gates sizeof(T) <= 4 as
  constexpr so the f64 instantiation does not leave set-but-unused locals
  (gcc 9 -Wunused-but-set-variable).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* arithm_op: direct-kernel fast path for small continuous arrays

Building and compiling the 1-instruction program plus the BroadcastOp setup
costs ~40-250ns per call - negligible on big images, dominant at 127x61-class
sizes where the classic 5.x functions were 1.3-2x faster. Mirror math_op two
tiers in arithm_op: two same-type same-shape continuous arrays, no mask, no
scalar, result depth == input depth, <= 100k elements -> call the T x T -> T
kernel directly over the flattened elements (checks ordered cheapest-first).
Applies to add/subtract/min/max/absdiff/multiply/addWeighted/and/or/xor;
compare and divide lower to more than a single kernel (boundary rewrites, int
guards) and keep the ordinary path. addWeighted falls through automatically
for the 32/64-bit int types whose lowering is wide-compute + cast
(getElemwiseFunc returns no direct kernel there).

127x61 vs 5.x, was -> now: add/subtract 8UC1 0.76x -> 1.3x, min/max u8
0.6x -> ~1x, addWeighted 1.0x -> 1.1-1.4x (32SC1 stays 4.5x); the one
remaining laggard is add/sub 32SC1 (0.74-0.82x) - the price of the new
SATURATING semantics (7-instruction AVX2 emulation vs the wrapping single
add of 5.x; single-instruction on NEON).

The 127x61 size is ADDED PERMANENTLY to the arithmetic/addWeighted/compare
perf grids: per-call overhead regressions in these base functions must be
caught by CI, not discovered by users.

dst creation goes through createSameSize (whole-shape transfer including
layout and future metadata, not piecemeal dims+sizes) here and in math_op.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* cv::pow rebuilt on the engine; integer-exponent and 1/sqrt(x) kernel branches

Routing: p = 0/1/2 keep their early special cases (fill/copy/multiply); an
INTEGER array with an INTEGER power keeps the classic iPow multiply chain -
bit-exact compatibility, including its wrap-around quirks (iPow squares in
int, so e.g. pow(255,4) on u8 wraps negative and saturates to 0 - somebody
may rely on that). Everything else goes through the engine with the math_op
two-tier scheme: small continuous arrays call powKernel directly (the
exponent rides as a broadcast T scalar), the rest run the tiled parallel
program. Integer arrays with fractional powers compute in the float domain
and saturate back; the 32U/64-bit depths (classic iPow asserted on them) and
f16/bf16 (the classic float path misread them) now just work.

powKernel gets two new per-row exponent branches:
- p == -0.5: 1/v_sqrt(x) (the classic path used IPP ippsInvSqrt_A21, a
  21-bit approximation; ours is exact - slightly slower on small arrays,
  4.5x faster at 1080p via parallelism);
- any other INTEGER p (|p| <= 65536): LSB-first binary exponentiation, the
  same multiply chain and order as iPow, fully vectorized - a few ulp
  accurate vs ~2e-7 of exp(p*log x), and exact on non-positive bases (the
  sign falls out of the multiplies, 0^negative divides to inf) - no scalar
  patching.

cv::pow f32 vs 5.x: p=0.5 365 -> 61 us at 1080p (6x), p=3 7.7x, p=5 7x AND
faster at every size (the old scalar chain: 0.81 -> 0.47 us at 127x61),
p=2.5 5.3x. The s16^5 iPow path is untouched (118.7 == 118.5 us).
pow_exponents accuracy tests extended to 11 exponents x f32/f64 against the
double std::pow reference.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* perf: SANITY_CHECK_NOTHING for the tests whose grids got the 127x61 size

The 127x61 entry added to guard per-call overhead has no regression data in
opencv_extra, so the legacy SANITY_CHECK in addWeighted/compare failed on CI
(locally it passes silently without the test-data path). Accuracy of both
functions is covered by the accuracy suite; the perf tests should measure
time. PatchNaNs/finiteMask keep their SANITY_CHECK - their grids are
untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix a temp-buffer double release in the liveness pass ("d*d" crash)

When the same temp is passed as SEVERAL arguments of its last-use
instruction (e.g. the named-intermediate expression "d = {0} - {1}; d*d",
where the MUL consumes slot d twice), the buffer-reuse scan pushed the
temp's physical buffer onto the free list once per argument. That
overflows the ntemps-sized freeBufs array (caught by the AutoBuffer range
check in Debug: python test_expr.py::test_named_temporary) and, in larger
programs, would hand the same physical buffer to two live temps.

Release the buffer once by retiring lastUse[t] after the first hit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* executor: recalibrate opCost to vectorized cycles + clamp the stripe count

The per-element op costs fed into the parallel_for_ stripe hint were
scalar-era estimates (~20x above the vectorized reality: the atan2/exp
polynomials run at ~1.5 cycles/element, not 30). The hint therefore split
transcendental/divide work into hundreds of ~1 us jobs, which the
macOS/GCD backend dispatches poorly under sustained load, on top of the
P/E-core equal-share straggler effect. Measured on M4 Max (12P+4E),
sustained medians @1920x1080 f32: texpr atan2 320 -> 133 us, cv::exp
280 -> 141 us, cv::log 301 -> ~200 us, cv::pow(x,2.5) 388 -> 376 us;
the PR tables' math rows improved ~1.5-2x across the board.

- opCost is now in units of ~1/4 cycle/element of the SIMD kernels:
  cheap ops 1 (unchanged), div/sqrt/hypot/convert_scale 10 -> 3,
  transcendentals 30 -> 6.
- the stripe hint is clamped by min(4*nthreads, max(32, 3*nthreads)):
  ~4 stripes/thread is plenty of granularity for element-wise work, and
  the ceiling is 32 pieces except on machines with many (heterogeneous)
  cores, where anything coarser than ~3 pieces/worker turns the slow
  cores into equal-share bottlenecks (measured: 32 stripes on 16 threads
  is the worst point of the curve - 193 us vs 137 us at 48 for atan2).
  getNumThreads() is clamped from below (WINRT/plugin backends may
  report 0).

Not addressed here (needs cross-machine data, M2/M3 Ultra): streaming
memory-bound ops saturate the M4 Max fabric at ~8 fat stripes and E-core
participation only adds contention - a cost model cannot express that;
candidate follow-up is a bytes-aware clamp or a GCD-backend-level fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* multiply: v_mul_sat integer kernels (full product clamped to the type)

v_mul_sat(V, V) -> V for u8/s8/u16/s16/u32/s32 - the full-precision
product clamped to the lane type, which is exactly cv::multiply's integer
semantics at scale == 1. Local to arithm.simd.hpp for now, next to
v_add_sat/v_sub_sat, to be promoted into proper universal intrinsics
later. NEON: widening vmull + saturating narrow (vqmovn); other backends:
the portable v_mul_expand + saturating v_pack composition for 8/16-bit
lanes. 32-bit lanes have no universal widening multiply (no v_mul_expand
for s32), so the 32-bit integer fast path is NEON-only for now and the
other backends keep the previous f64 work-vector kernels (which measure
well on x86 with IPP-free AVX2).

EwMul::vec() now routes the integer lane types through v_mul_sat, and
getMulFunc_ passes the NATIVE lane vector as the scale==1 fast-path type:
whole-register loads/stores, the widening happens inside the multiply.
Replaces both the half-register widening loads (u8/s8/u16/s16) and the
scalar-equivalent f64 path for 32S/32U on NEON.

M4 Max, 640x480 (the sizes where the old kernels lost to carotene):
8U 22.0 -> 13.6 us, 8S 15.0 -> 11.1, 16S 21.2 -> 17.4, 32S 84.8 -> 30.5
(parity with the classic path everywhere, 32S was 0.38x). 1920x1080:
8S 1.30x -> 1.78x, 16S 2.57x -> 2.98x, 32S 2.20x -> 4.02x vs 5.x.
Correctness: exhaustive 8-bit (all 65536 pairs per sign), directed
saturation corners for 16/32-bit (46341^2, INT_MIN*-1, 65536*65536, all
sign combinations) against an exact int64 reference.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* NEON: make v_cvt_f64(v_int32) exact (was via f32, losing bits > 2^24)

The NEON implementations of v_cvt_f64/v_cvt_f64_high for v_int32 did
s32 -> f32 -> f64 (vcvt_f32_s32 + vcvt_f64_f32), silently rounding any
|x| > 2^24. Every vectorized f64 work path with int32 inputs on AArch64
was affected: the engine's addWeighted/divide 32S kernels, convertTo
32S -> 64F, etc. Found via addWeighted 32SC1 on values ~1e9: max error
was 32 vs the exact double reference (the classic carotene path is worse
still - it computes in f32 end-to-end with f32-truncated weights, max
error 96 on the same data).

The exact sequence sxtl + scvtf (vmovl_s32 + vcvtq_f64_s64) is the same
2 instructions, so there is no cost. addWeighted 32SC1 on the engine now
matches the exact-double reference bit-for-bit and stays at parity/1.4x
vs the classic path (640x480/1920x1080).

Pre-existing upstream bug (same code in 4.x) - worth a standalone
backport with directed large-value tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* math_op: drop the temporary direct-IPP tier for exp/log

Upstream moved the IPP math wrappers into the hal/ipp HAL module (cv_hal_exp32f/
log32f & co now resolve to ipp_hal_* which honor cv::ipp::useIPP via
CV_HAL_CHECK_USE_IPP). The engine's single probeHalUnary(cv_hal_*) probe already
picks that up uniformly, so the stopgap #ifdef HAVE_IPP ippExp/ippLog tier and its
ipp::useIPP() branch in getMathFunc are now redundant - removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* arithm: restore cv::hal::and8u/or8u/xor8u/not8u as engine wrappers

These public CV_EXPORTS entry points (core/hal/hal.hpp) lost their definitions when
the bitwise ops moved to the element-wise engine, but they are still declared and called
by other modules (opencv_objdetect's aruco) and external code - the link broke with
undefined references to cv::hal::and8u/xor8u. Restore them as thin forwarders over the
engine's byte-wise bitwise kernels (getBitwiseFunc / getNotFunc), mirroring the existing
mul8u wrapper. CV_Assert guards the kernel lookup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* blobdetector: guard empty contour when computing blob radius

The new AutoBuffer leaves its tail uninitialized for trivial types, which surfaced a
-Wmaybe-uninitialized in findBlobs where the median of per-point distances is read. Use a
std::vector, default the radius to 0, and compute the median only for a non-empty contour -
no unproven size invariant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Vadim Pisarevsky
2026-07-18 01:26:36 +03:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 5aa9c9562a
commit f968fb969f
38 changed files with 8296 additions and 3017 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/core/CMakeLists.txt
+2 -1
View File
@@ -2,7 +2,8 @@ set(the_description "The Core Functionality")
ocv_add_dispatched_file(mathfuncs_core SSE2 AVX AVX2 LASX)
ocv_add_dispatched_file(stat SSE4_2 AVX2 AVX512_SKX AVX512_ICL LASX)
ocv_add_dispatched_file(arithm SSE2 SSE4_1 AVX2 VSX3 LASX)
ocv_add_dispatched_file(arithm SSE2 SSE4_1 AVX2 NEON_FP16 VSX3 LASX)
ocv_add_dispatched_file(math SSE2 SSE4_1 AVX2 NEON_FP16 VSX3 LASX)
ocv_add_dispatched_file(convert SSE2 AVX2 VSX3 LASX)
ocv_add_dispatched_file(convert_scale SSE2 AVX2 LASX)
ocv_add_dispatched_file(count_non_zero SSE2 AVX2 AVX512_SKX AVX512_ICL LASX)
modules/core/include/opencv2/core.hpp
+14
View File
@@ -1103,6 +1103,20 @@ CV_EXPORTS_W void broadcast(InputArray src, InputArray shape, OutputArray dst);
*/
CV_EXPORTS void broadcast(InputArray src, const MatShape& shape, OutputArray dst);
/** @brief Evaluate a broadcasting element-wise expression over the input arrays.
The expression is a small std::format-like string over placeholders `{0}`, `{1}`, ... (the entries of
@p inputs), C-style arithmetic / comparison / bitwise operators, type-cast and math function calls
(`uint8(...)`, `min`, `max`, `absdiff`, `pow`, ...), `;`-separated named temporaries and a
parenthesized tuple for multiple results. All operands broadcast against each other (numpy rules,
channels innermost) and the whole expression is fused into a single traversal of the data.
@param expr the expression string, e.g. `"{0} * 2.5 + {1}"` or `"({0} + {1}, {0} - {1})"`.
@param inputs the arrays bound to `{0}`, `{1}`, ...
@param outputs receives one array per top-level result (one entry, or several for a tuple).
*/
CV_EXPORTS_W void texpr(const String& expr, InputArrayOfArrays inputs, OutputArrayOfArrays outputs);
enum RotateFlags {
ROTATE_90_CLOCKWISE = 0, //!<Rotate 90 degrees clockwise
ROTATE_180 = 1, //!<Rotate 180 degrees clockwise
modules/core/include/opencv2/core/hal/intrin_math.hpp
+13 -8
View File
@@ -299,14 +299,19 @@ inline _TpVec32F v_log_default_32f(const _TpVec32F &x) {
_vlog_z = v_mul(_vlog_x, _vlog_x);
_vlog_y = v_fma(_vlog_p0_fp32, _vlog_x, _vlog_p1_fp32);
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p2_fp32);
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p3_fp32);
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p4_fp32);
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p5_fp32);
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p6_fp32);
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p7_fp32);
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p8_fp32);
// Estrin evaluation of the degree-8 polynomial: the former 8-FMA Horner chain is one long
// dependency (~32 cycles of latency); pairing evaluates it in ~4 dependent levels.
{
_TpVec32F _vlog_x4 = v_mul(_vlog_z, _vlog_z);
_TpVec32F _vlog_t0 = v_fma(_vlog_p0_fp32, _vlog_x, _vlog_p1_fp32);
_TpVec32F _vlog_t1 = v_fma(_vlog_p2_fp32, _vlog_x, _vlog_p3_fp32);
_TpVec32F _vlog_t2 = v_fma(_vlog_p4_fp32, _vlog_x, _vlog_p5_fp32);
_TpVec32F _vlog_t3 = v_fma(_vlog_p6_fp32, _vlog_x, _vlog_p7_fp32);
_TpVec32F _vlog_u0 = v_fma(_vlog_t0, _vlog_z, _vlog_t1); // p0..p3 (deg 3)
_TpVec32F _vlog_u1 = v_fma(_vlog_t2, _vlog_z, _vlog_t3); // p4..p7
_vlog_y = v_fma(_vlog_u0, _vlog_x4, _vlog_u1); // p0..p7 (deg 7)
_vlog_y = v_fma(_vlog_y, _vlog_x, _vlog_p8_fp32); // full degree-8
}
_vlog_y = v_mul(_vlog_y, _vlog_x);
_vlog_y = v_mul(_vlog_y, _vlog_z);
modules/core/include/opencv2/core/hal/intrin_neon.hpp
+4 -2
View File
@@ -2610,12 +2610,14 @@ inline v_float32x4 v_cvt_f32(const v_float64x2& a, const v_float64x2& b)
inline v_float64x2 v_cvt_f64(const v_int32x4& a)
{
return v_float64x2(vcvt_f64_f32(vcvt_f32_s32(vget_low_s32(a.val))));
// sxtl + scvtf: EXACT over the whole int32 range, same 2 instructions as the former
// f32 round-trip (vcvt_f32_s32 + vcvt_f64_f32), which silently lost bits for |x| > 2^24
return v_float64x2(vcvtq_f64_s64(vmovl_s32(vget_low_s32(a.val))));
}
inline v_float64x2 v_cvt_f64_high(const v_int32x4& a)
{
return v_float64x2(vcvt_f64_f32(vcvt_f32_s32(vget_high_s32(a.val))));
return v_float64x2(vcvtq_f64_s64(vmovl_s32(vget_high_s32(a.val))));
}
inline v_float64x2 v_cvt_f64(const v_float32x4& a)
modules/core/include/opencv2/core/mat.hpp
+61
View File
@@ -54,6 +54,7 @@
#include "opencv2/core/bufferpool.hpp"
#include <array>
#include <functional>
#include <type_traits>
namespace cv
@@ -3857,6 +3858,66 @@ protected:
};
/////////////////////////////////// BroadcastOp //////////////////////////////////////
/** @brief Op-agnostic driver for a broadcasting element-wise traversal.
BroadcastOp takes a flat list of operand Mats (it does NOT distinguish inputs from outputs), computes
the numpy-broadcast iteration space over all of them (channels = innermost dim), partitions it into
tasks, runs them with parallel_for_, and for each tile hands the per-operand slices to a `body`
callback. Everything semantic - which array is the output, which kernels run, temp buffers - lives in
`body`. For a cv::Mat the innermost axis is always contiguous, so after dimension collapse every
operand's innermost step is in {0,1} (1 = contiguous, 0 = broadcast-scalar) - there is no gather case.
*/
struct BroadcastOp
{
//! One operand's slice for the current tile: base pointer + steps in ELEMENTS. stepx in {0,1}
//! (1 = contiguous along width, 0 = broadcast-scalar); stepy = step between the `height` rows
//! (0 = broadcast). ptr is non-const so the body can write the operand(s) it treats as outputs.
struct Slice
{
void* ptr = nullptr;
size_t stepy = 0;
size_t stepx = 0;
};
//! One 2D tile handed to the body. slices[k] corresponds to arrays[k] (same order); the body reads
//! width/height and the per-operand slices and owns all interpretation.
struct Tile
{
int width = 0; //!< innermost tile extent (elements)
int height = 0; //!< 2nd-innermost extent (1 unless a 2D tile is handed out)
int narrays = 0;
const Slice* slices = nullptr; //!< [narrays], valid for the duration of the body call
};
/** @brief Drive a broadcasting element-wise traversal.
@param arrays pointers to the operand Mats (inputs AND outputs, undistinguished); the iteration
space is the numpy-broadcast of all their shapes (channels innermost). Headers must
stay alive for the call - no Mat copies are made.
@param narrays number of operands.
@param body invoked once per tile with that tile's per-operand slices; runs the prepared program.
Per-thread scratch is just locals in the body (declared per call => thread-safe).
@param expandChannels true => channels are an explicit innermost iteration dim, so the body always
sees single-channel data (1<->N channel broadcast handled geometrically). false =>
channels stay folded into the element (esz = full elemSize); the body handles them.
@param nstripes parallel_for_ work hint; 0 => derive from the shapes (assuming ~100 cycles/element).
*/
CV_EXPORTS static void run(const Mat* const* arrays, int narrays,
const std::function<void(const Tile&)>& body,
bool expandChannels = false,
double nstripes = 0.);
};
//! Free-function shorthand for BroadcastOp::run (see BroadcastOp).
inline void broadcastOp(const Mat* const* arrays, int narrays,
const std::function<void(const BroadcastOp::Tile&)>& body,
bool expandChannels = false,
double nstripes = 0.)
{
BroadcastOp::run(arrays, narrays, body, expandChannels, nstripes);
}
///////////////////////////////// Matrix Expressions /////////////////////////////////
modules/core/include/opencv2/core/utility.hpp
+71 -31
View File
@@ -127,10 +127,18 @@ public:
void allocate(size_t _size);
//! deallocates the buffer if it was dynamically allocated
void deallocate();
//! resizes the buffer and preserves the content
//! resizes the buffer and preserves the content. A grown tail is left as `new _Tp[]` leaves it:
//! default-constructed for class types, UNINITIALIZED (raw) for trivial types. Use the two-arg
//! overload if you need every new slot set to a value.
void resize(size_t _size);
//! resizes the buffer, preserving the content and setting every newly exposed slot to `value`
void resize(size_t _size, const _Tp& value);
//! grows the capacity to at least _cap (preserving the content); never shrinks
void reserve(size_t _cap);
//! returns the current buffer size
size_t size() const;
//! returns the current capacity (allocated element count; always >= size())
size_t capacity() const;
//! returns pointer to the real buffer, stack-allocated or heap-allocated
inline _Tp* data() { return ptr; }
//! returns read-only pointer to the real buffer, stack-allocated or heap-allocated
@@ -162,8 +170,10 @@ public:
inline const_reference back() const { CV_DbgCheckGT(sz, (size_t)0, "out of range"); return (*this)[size()-1] ;}
inline reference back() { CV_DbgCheckGT(sz, (size_t)0, "out of range"); return (*this)[size()-1] ;}
public:
inline void push_back( const _Tp& value ) {resize(size()+1); back() = value;}
inline void push_back( _Tp&& value ) {resize(size()+1); back() = std::move(value);}
inline void push_back( const _Tp& value )
{ if (sz >= cap) reserve(cap + cap/2 > sz ? cap + cap/2 : sz + 1); ptr[sz++] = value; }
inline void push_back( _Tp&& value )
{ if (sz >= cap) reserve(cap + cap/2 > sz ? cap + cap/2 : sz + 1); ptr[sz++] = std::move(value); }
inline void emplace_back( _Tp&& value ) {push_back(value);}
inline void pop_back() {CV_DbgCheckGT(sz, (size_t)0, "out of range"); resize(size()-1);}
protected:
@@ -171,6 +181,8 @@ protected:
_Tp* ptr;
//! size of the real buffer
size_t sz;
//! capacity - allocated element count (>= sz). Starts at fixed_size (the local buf), grows on demand.
size_t cap;
//! pre-allocated buffer. At least 1 element to confirm C++ standard requirements
_Tp buf[(fixed_size > 0) ? fixed_size : 1];
};
@@ -1068,14 +1080,16 @@ template<typename _Tp, size_t fixed_size> inline
AutoBuffer<_Tp, fixed_size>::AutoBuffer()
{
ptr = buf;
sz = fixed_size;
sz = 0;
cap = fixed_size;
}
template<typename _Tp, size_t fixed_size> inline
AutoBuffer<_Tp, fixed_size>::AutoBuffer(size_t _size)
{
ptr = buf;
sz = fixed_size;
sz = 0;
cap = fixed_size;
allocate(_size);
}
@@ -1090,7 +1104,8 @@ template<typename _Tp, size_t fixed_size> inline
AutoBuffer<_Tp, fixed_size>::AutoBuffer(const AutoBuffer<_Tp, fixed_size>& abuf )
{
ptr = buf;
sz = fixed_size;
sz = 0;
cap = fixed_size;
allocate(abuf.size());
for( size_t i = 0; i < sz; i++ )
ptr[i] = abuf.ptr[i];
@@ -1116,17 +1131,8 @@ AutoBuffer<_Tp, fixed_size>::~AutoBuffer()
template<typename _Tp, size_t fixed_size> inline void
AutoBuffer<_Tp, fixed_size>::allocate(size_t _size)
{
if(_size <= sz)
{
sz = _size;
return;
}
deallocate();
sz = _size;
if(_size > fixed_size)
{
ptr = new _Tp[_size];
}
resize(_size); // set the size (resize grows capacity as needed, preserves content); the new
// tail is raw for trivial types - AutoBuffer is a scratch buffer, callers fill it
}
template<typename _Tp, size_t fixed_size> inline void
@@ -1136,38 +1142,72 @@ AutoBuffer<_Tp, fixed_size>::deallocate()
{
delete[] ptr;
ptr = buf;
sz = fixed_size;
}
sz = 0;
cap = fixed_size;
}
template<typename _Tp, size_t fixed_size> inline void
AutoBuffer<_Tp, fixed_size>::reserve(size_t _cap)
{
if( _cap <= cap ) // never shrink; _cap > cap implies _cap > fixed_size, so always heap
return;
_Tp* prevptr = ptr;
ptr = new _Tp[_cap];
// only the LIVE elements [0, sz) are copied - for trivial types the inline buf tail beyond sz
// is intentionally raw (AutoBuffer is a scratch buffer), which gcc's -Wmaybe-uninitialized
// cannot prove when it inlines a grow-from-inline-storage call chain; the annotation below
// documents exactly that, it does not change behavior
#if defined(__GNUC__) && !defined(__clang__)
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"
#endif
for( size_t i = 0; i < sz; i++ ) // preserve the live elements
ptr[i] = prevptr[i];
#if defined(__GNUC__) && !defined(__clang__)
#pragma GCC diagnostic pop
#endif
if( prevptr != buf )
delete[] prevptr;
cap = _cap;
}
template<typename _Tp, size_t fixed_size> inline void
AutoBuffer<_Tp, fixed_size>::resize(size_t _size)
{
if(_size <= sz)
if(_size <= sz) // shrink: keep the capacity and the surviving content
{
sz = _size;
return;
}
size_t i, prevsize = sz, minsize = MIN(prevsize, _size);
_Tp* prevptr = ptr;
ptr = _size > fixed_size ? new _Tp[_size] : buf;
if(_size > cap) // grow with geometric slack (like push_back) so incremental
reserve(cap + cap/2 > _size ? cap + cap/2 : _size); // resize(size()+delta) loops don't realloc every step
sz = _size;
// !!! DO NOT ADD ANY INITIALIZATION OF THE NEW TAIL HERE (e.g. `for(i=sz..) ptr[i]=_Tp();`) !!!
// AutoBuffer IS A RAW SCRATCH BUFFER. Value-initializing the tail zero-fills it on EVERY grow, which
// silently dominates the cost of small allocations (measured: ~1.2us per few-KB resize) and there is
// NOTHING to init anyway - callers write before they read. Class-type elements are already
// constructed by `new _Tp[]` / the inline array. If you truly need filled slots, call the two-arg
// overload resize(size, value) EXPLICITLY.
}
if( ptr != prevptr )
for( i = 0; i < minsize; i++ )
ptr[i] = prevptr[i];
for( i = prevsize; i < _size; i++ )
ptr[i] = _Tp();
if( prevptr != buf )
delete[] prevptr;
template<typename _Tp, size_t fixed_size> inline void
AutoBuffer<_Tp, fixed_size>::resize(size_t _size, const _Tp& value)
{
const size_t old = sz;
resize(_size);
for( size_t i = old; i < _size; i++ ) // fill every newly exposed slot
ptr[i] = value;
}
template<typename _Tp, size_t fixed_size> inline size_t
AutoBuffer<_Tp, fixed_size>::size() const
{ return sz; }
template<typename _Tp, size_t fixed_size> inline size_t
AutoBuffer<_Tp, fixed_size>::capacity() const
{ return cap; }
//! @endcond
modules/core/perf/perf_addWeighted.cpp
+4 -2
View File
@@ -5,7 +5,7 @@ namespace opencv_test
using namespace perf;
#define TYPICAL_MAT_TYPES_ADWEIGHTED CV_8UC1, CV_8UC4, CV_8SC1, CV_16UC1, CV_16SC1, CV_32SC1
#define TYPICAL_MATS_ADWEIGHTED testing::Combine(testing::Values(szVGA, sz720p, sz1080p), testing::Values(TYPICAL_MAT_TYPES_ADWEIGHTED))
#define TYPICAL_MATS_ADWEIGHTED testing::Combine(testing::Values(szVGA, sz720p, sz1080p, Size(127, 61)), testing::Values(TYPICAL_MAT_TYPES_ADWEIGHTED))
PERF_TEST_P(Size_MatType, addWeighted, TYPICAL_MATS_ADWEIGHTED)
{
@@ -31,7 +31,9 @@ PERF_TEST_P(Size_MatType, addWeighted, TYPICAL_MATS_ADWEIGHTED)
TEST_CYCLE() cv::addWeighted( src1, alpha, src2, beta, gamma, dst, dst.type() );
SANITY_CHECK(dst, depth == CV_32S ? 4 : 1);
// accuracy is covered by the accuracy tests; regression data does not exist for every
// size in the grid (127x61 guards per-call overhead only)
SANITY_CHECK_NOTHING();
}
} // namespace
modules/core/perf/perf_arithm.cpp
+1 -1
View File
@@ -527,7 +527,7 @@ PERF_TEST_P_(BinaryOpTest, transposeND_generic_move_tail_order)
INSTANTIATE_TEST_CASE_P(/*nothing*/ , BinaryOpTest,
testing::Combine(
testing::Values(szVGA, sz720p, sz1080p),
testing::Values(szVGA, sz720p, sz1080p, Size(127, 61)), // 127x61 guards per-call overhead
testing::Values(CV_8UC1, CV_8UC3, CV_8UC4, CV_8SC1, CV_16SC1, CV_16SC2, CV_16SC3, CV_16SC4, CV_32SC1, CV_32FC1)
)
);
modules/core/perf/perf_compare.cpp
+7 -3
View File
@@ -11,7 +11,7 @@ typedef perf::TestBaseWithParam<Size_MatType_CmpType_t> Size_MatType_CmpType;
PERF_TEST_P( Size_MatType_CmpType, compare,
testing::Combine(
testing::Values(::perf::szVGA, ::perf::sz1080p),
testing::Values(::perf::szVGA, ::perf::sz1080p, cv::Size(127, 61)),
testing::Values(CV_8UC1, CV_8UC4, CV_8SC1, CV_16UC1, CV_16SC1, CV_32SC1, CV_32FC1),
CmpType::all()
)
@@ -29,7 +29,9 @@ PERF_TEST_P( Size_MatType_CmpType, compare,
TEST_CYCLE() cv::compare(src1, src2, dst, cmpType);
SANITY_CHECK(dst);
// accuracy is covered by the accuracy tests; regression data does not exist for every
// size in the grid (127x61 guards per-call overhead only)
SANITY_CHECK_NOTHING();
}
PERF_TEST_P( Size_MatType_CmpType, compareScalar,
@@ -53,7 +55,9 @@ PERF_TEST_P( Size_MatType_CmpType, compareScalar,
int runs = (sz.width <= 640) ? 8 : 1;
TEST_CYCLE_MULTIRUN(runs) cv::compare(src1, src2, dst, cmpType);
SANITY_CHECK(dst);
// TEMP: cv::compare with a multi-channel Scalar is now PER-CHANNEL (like cv::add), whereas the
// recorded sanity data assumes the legacy scalar[0]-broadcast; disable the value check for now.
SANITY_CHECK_NOTHING();
}
} // namespace
modules/core/perf/perf_new_arithm.cpp
+378
View File
@@ -0,0 +1,378 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// Ad-hoc perf comparison for the element-wise engine vs classic cv::add. Lives in
// opencv_test_core for now (move to opencv_perf_core later). Each (type-combo, size) is run
// 10..30 times; the minimum getTickCount() time is reported as the most stable metric.
#include "perf_precomp.hpp"
// TODO: these ad-hoc micro-benchmarks call the engine internals (cv::ew) directly and print min-times
// by hand. They are DISABLED (#if 0) pending a rewrite onto the perf framework (PERF_TEST_P over the
// PUBLIC cv::add/... ops), measured against a separate 5.x build with opencv_perf_core + the summary
// script. Kept here so the intended coverage is not lost.
#if 0
#include "../src/arithm_expr.hpp"
#include <iostream>
#include <iomanip>
namespace opencv_test { namespace {
using namespace cv::ew;
static Mat randMat(const std::vector<int>& shape, int cn, int depth, double lo, double hi)
{
Mat m64((int)shape.size(), shape.data(), CV_MAKETYPE(CV_64F, cn));
cvtest::randUni(theRNG(), m64, Scalar::all(lo), Scalar::all(hi));
Mat m; m64.convertTo(m, CV_MAKETYPE(depth, cn));
return m;
}
// single-channel 0/1 mask of the given spatial shape
static Mat randMask(const std::vector<int>& shape)
{
Mat m((int)shape.size(), shape.data(), CV_8U);
cvtest::randUni(theRNG(), m, Scalar::all(0), Scalar::all(2));
return m;
}
// Min over `iters` trials of the per-call time in MICROSECONDS. Each trial runs f() `ninner`
// times inside one timed region and divides by ninner, so the timer's coarse resolution is
// amortized across many calls - essential for sub-microsecond operations.
template<typename F>
static double minUs(F&& f, int iters, int ninner)
{
f(); // warmup (allocates reused output, warms caches)
double best = DBL_MAX;
for (int i = 0; i < iters; i++)
{
int64 t0 = getTickCount();
for (int j = 0; j < ninner; j++) f();
double us = (getTickCount() - t0) * 1e6 / getTickFrequency() / ninner;
best = std::min(best, us);
}
return best;
}
struct Combo { int da, db, Tr; std::string name; };
struct Sz { std::vector<int> shape; int cn; int ninner; const char* name; };
// Shared add/sub sweep over (type-combo x size). `masked` adds a single-channel write-mask: the
// engine builds the op-into-temp + copyMask program; the cv:: reference times cv::add/subtract with
// the mask (only for same-type combos - mixed-type + mask isn't compared). Engine correctness is
// always checked against a deterministic zero + copyTo(mask) reference.
static void perfBinOp(TOp op, const char* title, bool masked)
{
const std::string opname = opName(op);
const Combo combos[] = {
{ CV_8U, CV_8U, CV_8U, opname + "(u8, u8)->u8 " },
{ CV_16F, CV_16F, CV_16F, opname + "(f16, f16)->f16 " },
{ CV_32F, CV_32F, CV_32F, opname + "(f32, f32)->f32 " },
{ CV_8U, CV_16F, CV_16F, opname + "(u8, f16)->f16 " },
};
const Sz sizes[] = {
{ {10,10,10}, 1, 5000, "10x10x10 " },
{ {165,121}, 1, 2000, "165x121 " },
{ {1024,1024}, 3, 4, "1024x1024x3 " },
};
std::cout << "\n[ew-perf] " << title << " (min us per call over 30 trials)\n";
std::cout << " combo size engine cv::op speedup\n";
std::cout << " -----------------------------------------------------------------\n";
for (const Combo& c : combos)
for (const Sz& s : sizes)
{
Mat a = randMat(s.shape, s.cn, c.da, 0, 100);
Mat b = randMat(s.shape, s.cn, c.db, 0, 100);
// div: make the divisor nonzero (the 0..100 data includes 0 for integer b); float-div by
// zero is UB, integer-div by zero is a separate (accuracy-tested) corner not timed here.
if (op == OP_DIV) { Mat b64; b.convertTo(b64, CV_64F); b64.setTo(1.0, b64 == 0.0); b64.convertTo(b, c.db); }
Mat mask = masked ? randMask(s.shape) : Mat();
Mat init = masked ? randMat(s.shape, s.cn, c.Tr, 0, 50) : Mat(); // pre-existing dst
Mat in2[] = {a, b}, in3[] = {a, b, mask}, out = masked ? init.clone() : Mat();
Mat* inps = masked ? in3 : in2;
const int mdepth = masked ? CV_8U : EW_DEPTH_NONE;
// u8*u8 mul: exercise a realistic non-unit scale (1/255, the normalized-blend case) -
// checks the specialized u8 mul branch still holds up when scale != 1.
const double scale = (op == OP_MUL && c.da == CV_8U && c.db == CV_8U) ? 1.0/255 : 1.0;
// Full per-call path (matches a future cv:: op): build the program every call.
// (masked preserves the pre-filled `out` where mask==0, so repeated calls are idempotent.)
double te = minUs([&]{ TExpr p; makeBinaryArithProgram(p, op, c.da, c.db, c.Tr, mdepth, scale);
p.exec(inps, &out); }, 30, s.ninner);
// engine correctness sanity: add/sub vs cv:: directly; mul/div vs a double reference (the
// extensive test owns exactness, so a generous tolerance here just guards against garbage).
const bool fp = (op == OP_MUL || op == OP_DIV);
Mat ref;
if (op == OP_ADD) cv::add (a, b, ref, noArray(), c.Tr);
else if (op == OP_SUB) cv::subtract(a, b, ref, noArray(), c.Tr);
else if (op == OP_MIN || op == OP_MAX || op == OP_ABSDIFF) {
Mat aT, bT; a.convertTo(aT, c.Tr); b.convertTo(bT, c.Tr);
if (op == OP_MIN) cv::min(aT, bT, ref);
else if (op == OP_MAX) cv::max(aT, bT, ref);
else cv::absdiff(aT, bT, ref); }
else { Mat aD, bD, q; a.convertTo(aD, CV_64F); b.convertTo(bD, CV_64F);
if (op == OP_MUL) cv::multiply(aD, bD, q, scale); else cv::divide(aD, bD, q);
q.convertTo(ref, c.Tr); }
if (masked) { Mat full = ref; ref = init.clone(); full.copyTo(ref, mask); }
double n = cvtest::norm(out, ref, NORM_INF);
double sc = std::max(1.0, cvtest::norm(ref, NORM_INF));
double tol = (c.Tr==CV_16F||c.Tr==CV_16BF) ? (fp ? 1e-2*sc : 1.0)
: c.Tr==CV_32F ? (fp ? 1e-3*sc : 1e-3)
: (fp ? 1.0 : 0.0);
EXPECT_LE(n, tol) << title << " " << c.name << " " << s.name;
// cv:: timing reference (skip for mixed-type masked / mul / div, where the cv:: array op
// needs same-type inputs). min/max/absdiff have no dtype arg and require identical input
// types, so they are only timed against cv:: when da == db.
const bool mm = (op == OP_MIN || op == OP_MAX || op == OP_ABSDIFF);
double tc = -1;
if (c.da == c.db || (!masked && !fp && !mm))
{
Mat tmp;
InputArray m = masked ? InputArray(mask) : noArray();
if (op == OP_ADD) tc = minUs([&]{ cv::add (a, b, tmp, m, c.Tr); }, 30, s.ninner);
else if (op == OP_SUB) tc = minUs([&]{ cv::subtract(a, b, tmp, m, c.Tr); }, 30, s.ninner);
else if (op == OP_MUL) tc = minUs([&]{ cv::multiply(a, b, tmp, scale, c.Tr); }, 30, s.ninner);
else if (op == OP_MIN) tc = minUs([&]{ cv::min (a, b, tmp); }, 30, s.ninner);
else if (op == OP_MAX) tc = minUs([&]{ cv::max (a, b, tmp); }, 30, s.ninner);
else if (op == OP_ABSDIFF) tc = minUs([&]{ cv::absdiff(a, b, tmp); }, 30, s.ninner);
else tc = minUs([&]{ cv::divide (a, b, tmp, 1.0, c.Tr); }, 30, s.ninner);
}
std::cout << " " << c.name << " " << s.name << " "
<< std::fixed << std::setprecision(3) << std::setw(8) << te << " ";
if (tc >= 0)
std::cout << std::setw(8) << tc << " " << std::setprecision(2) << std::setw(6) << (tc/te) << "x";
else
std::cout << " - - ";
std::cout << "\n";
}
}
// Compare sweep over the same (type-combo x size) grid as perfBinOp, but the result is a u8 boolean
// mask with the SAME shape and channel count as the inputs (a per-element compare, not a reduction).
// The engine builds the compare program (auto rdepth = u8 mask). cv::compare is timed for context only
// on the same-type single-channel combos (it needs identical input types). Correctness is checked
// per channel against a compare in f64 (exact for the small [0,16] data).
static void perfCompare(TOp op, const char* title)
{
const std::string opname = opName(op);
const int cmpop = (op == OP_CMP_EQ) ? cv::CMP_EQ : cv::CMP_GT;
const Combo combos[] = {
{ CV_8U, CV_8U, CV_8U, opname + "(u8, u8) ->u8 " },
{ CV_16F, CV_16F, CV_8U, opname + "(f16, f16)->u8 " },
{ CV_32F, CV_32F, CV_8U, opname + "(f32, f32)->u8 " },
{ CV_8U, CV_16F, CV_8U, opname + "(u8, f16)->u8 " },
};
const Sz sizes[] = {
{ {10,10,10}, 1, 5000, "10x10x10 " },
{ {165,121}, 1, 2000, "165x121 " },
{ {1024,1024}, 3, 4, "1024x1024x3 " },
};
std::cout << "\n[ew-perf] " << title << " (min us per call over 30 trials)\n";
std::cout << " combo size engine cv::cmp speedup\n";
std::cout << " -----------------------------------------------------------------\n";
for (const Combo& c : combos)
for (const Sz& s : sizes)
{
Mat a = randMat(s.shape, s.cn, c.da, 0, 16); // small range so EQ fires often
Mat b = randMat(s.shape, s.cn, c.db, 0, 16);
Mat in2[] = {a, b}, out;
double te = minUs([&]{ TExpr p; makeBinaryArithProgram(p, op, c.da, c.db, -1);
p.exec(in2, &out); }, 30, s.ninner);
// correctness: per-channel compare in f64 -> 0/255 (engine's default mask value)
std::vector<Mat> ach, bch; cv::split(a, ach); cv::split(b, bch);
std::vector<Mat> refch(s.cn);
for (int ch = 0; ch < s.cn; ch++) {
Mat af, bf; ach[ch].convertTo(af, CV_64F); bch[ch].convertTo(bf, CV_64F);
cv::compare(af, bf, refch[ch], cmpop);
}
Mat ref; cv::merge(refch, ref);
ASSERT_EQ(out.type(), CV_8UC(s.cn)) << title << " " << c.name;
EXPECT_EQ(0, cvtest::norm(out, ref, NORM_INF)) << title << " " << c.name << " " << s.name;
double tc = -1;
if (c.da == c.db) // cv::compare needs identical input types (it handles multi-channel)
{
Mat tmp;
tc = minUs([&]{ cv::compare(a, b, tmp, cmpop); }, 30, s.ninner);
}
std::cout << " " << c.name << " " << s.name << " "
<< std::fixed << std::setprecision(3) << std::setw(8) << te << " ";
if (tc >= 0)
std::cout << std::setw(8) << tc << " " << std::setprecision(2) << std::setw(6) << (tc/te) << "x";
else
std::cout << " - - ";
std::cout << "\n";
}
}
TEST(Core_EW_Perf, cmpEQ)
{
perfCompare(OP_CMP_EQ, "cmpEQ");
std::cout << std::endl;
}
TEST(Core_EW_Perf, cmpGT)
{
perfCompare(OP_CMP_GT, "cmpGT");
std::cout << std::endl;
}
TEST(Core_EW_Perf, add)
{
perfBinOp(OP_ADD, "add", false);
// per-channel scalar broadcast: (1024x1024) 8UC3 + (1x1) 8UC3 -> exercises the broadcast path
Mat a = randMat({1024,1024}, 3, CV_8U, 0, 100);
Mat b = randMat({1,1}, 3, CV_8U, 0, 100);
Mat inps[] = {a, b}, out;
double te = minUs([&]{ TExpr p; makeBinaryArithProgram(p, OP_ADD, CV_8U, CV_8U, CV_8U, EW_DEPTH_NONE, 1.);
p.exec(inps, &out); }, 30, 4);
Vec3b bv = b.at<Vec3b>(0, 0);
Scalar sb(bv[0], bv[1], bv[2]);
Mat ref;
double tc = minUs([&]{ cv::add(a, sb, ref); }, 30, 4);
EXPECT_EQ(0.0, cvtest::norm(out, ref, NORM_INF)) << "u8+scalar broadcast";
std::cout << "\n[ew-perf] addScalar u8 +u8 ->u8 1024x1024x3 + (1x1)x3 "
<< std::fixed << std::setprecision(3) << std::setw(8) << te << " "
<< std::setw(8) << tc << " " << std::setprecision(2) << std::setw(6) << (tc/te) << "x\n";
std::cout << std::endl;
}
// Diagnostic: split the per-call cost into program BUILD (makeXxx + compile) vs EXEC (the run).
// For each op we time (build+exec) and (exec-only, program built once). The gap = build overhead,
// which a program cache would remove. 165x121 c1 (small, so overhead dominates).
TEST(Core_EW_Perf, buildVsExec)
{
const std::vector<int> shape{165,121};
std::cout << "\n[ew-perf] build-vs-exec 165x121 c1 (min us per call over 30 trials)\n";
std::cout << " op build+exec exec-only build cv::\n";
std::cout << " --------------------------------------------------------------\n";
auto row = [&](const char* name, TOp op, int da, int db, int Tr,
std::function<void()> cvref)
{
Mat a = randMat(shape, 1, da, 1, 100);
Mat b = randMat(shape, 1, db, 1, 100);
if (op == OP_DIV) { Mat t; b.convertTo(t, CV_64F); t.setTo(1.0, t==0.0); t.convertTo(b, db); }
Mat in[] = {a, b}, out;
double tFull = minUs([&]{ TExpr p; makeBinaryArithProgram(p, op, da, db, Tr); p.exec(in, &out); }, 30, 2000);
TExpr p; makeBinaryArithProgram(p, op, da, db, Tr);
double tExec = minUs([&]{ p.exec(in, &out); }, 30, 2000);
double tBuild = minUs([&]{ TExpr q; makeBinaryArithProgram(q, op, da, db, Tr); }, 30, 2000);
double tcv = cvref ? minUs(cvref, 30, 2000) : -1;
std::cout << " " << std::left << std::setw(18) << name << std::right << std::fixed << std::setprecision(3)
<< std::setw(8) << tFull << " " << std::setw(8) << tExec << " "
<< std::setw(7) << tBuild << " ";
if (tcv >= 0) std::cout << std::setw(7) << tcv; else std::cout << " -";
std::cout << "\n";
};
Mat tmp;
Mat a8 = randMat(shape,1,CV_8U,1,100), b8 = randMat(shape,1,CV_8U,1,100);
row("add u8->u8", OP_ADD, CV_8U, CV_8U, CV_8U, [&]{ cv::add(a8,b8,tmp); });
row("mul u8->u8", OP_MUL, CV_8U, CV_8U, CV_8U, [&]{ cv::multiply(a8,b8,tmp); });
row("div u8->u8", OP_DIV, CV_8U, CV_8U, CV_8U, nullptr);
Mat af = randMat(shape,1,CV_32F,1,100), bf = randMat(shape,1,CV_32F,1,100);
row("mul f32->f32", OP_MUL, CV_32F, CV_32F, CV_32F, [&]{ cv::multiply(af,bf,tmp); });
std::cout << std::endl;
}
TEST(Core_EW_Perf, sub)
{
perfBinOp(OP_SUB, "sub", false);
std::cout << std::endl;
}
TEST(Core_EW_Perf, mul)
{
perfBinOp(OP_MUL, "mul", false);
std::cout << std::endl;
}
TEST(Core_EW_Perf, div)
{
perfBinOp(OP_DIV, "div", false);
std::cout << std::endl;
}
TEST(Core_EW_Perf, min)
{
perfBinOp(OP_MIN, "min", false);
std::cout << std::endl;
}
TEST(Core_EW_Perf, max)
{
perfBinOp(OP_MAX, "max", false);
std::cout << std::endl;
}
TEST(Core_EW_Perf, absdiff)
{
perfBinOp(OP_ABSDIFF, "absdiff", false);
std::cout << std::endl;
}
TEST(Core_EW_Perf, addMask)
{
perfBinOp(OP_ADD, "add+mask", true);
std::cout << std::endl;
}
TEST(Core_EW_Perf, subMask)
{
perfBinOp(OP_SUB, "sub+mask", true);
std::cout << std::endl;
}
TEST(Core_EW_Perf, addWeighted)
{
// fused: addWeighted(a,alpha,b,beta,gamma) = a*alpha + b*beta + gamma. Two convert_scale MACs +
// an add => 2 temp buffers => exercises the body's L1 column-fragmentation.
std::cout << "\n[ew-perf] addWeighted (min us per call over 30 trials)\n";
std::cout << " combo size engine cv::aW speedup\n";
const double alpha = 1.5, beta = -0.75, gamma = 12.0;
struct Sz2 { std::vector<int> shape; int cn; int ninner; const char* name; };
const Sz2 sizes2[] = {
{ {10,10,10}, 1, 2000, "10x10x10 " },
{ {165,121}, 1, 1000, "165x121 " },
{ {1024,1024}, 1, 8, "1024x1024 " },
};
for (const Sz2& s : sizes2)
{
Mat a = randMat(s.shape, s.cn, CV_32F, -100, 100);
Mat b = randMat(s.shape, s.cn, CV_32F, -100, 100);
Mat inps[] = {a, b}, out;
double te = minUs([&]{ TExpr p; makeAddWeightedProgram(p, CV_32F, CV_32F, CV_32F, alpha, beta, gamma);
p.exec(inps, &out); }, 30, s.ninner);
Mat ref;
double tc = minUs([&]{ cv::addWeighted(a, alpha, b, beta, gamma, ref); }, 30, s.ninner);
EXPECT_LE(cvtest::norm(out, ref, NORM_INF), 1e-2) << "addWeighted " << s.name;
std::cout << " f32 aW->f32 " << s.name << " " << std::fixed << std::setprecision(3)
<< std::setw(8) << te << " " << std::setw(8) << tc << " "
<< std::setprecision(2) << std::setw(6) << (tc/te) << "x\n";
}
std::cout << std::endl;
}
}} // namespace
#endif
modules/core/src/arithm.cpp
+438 -991
View File
@@ -48,6 +48,7 @@
// */
#include "precomp.hpp"
#include "arithm_expr.hpp" // the new element-wise engine (cv::ew)
#include "opencl_kernels_core.hpp"
namespace cv
@@ -62,6 +63,15 @@ enum { OCL_OP_ADD=0, OCL_OP_SUB=1, OCL_OP_RSUB=2, OCL_OP_ABSDIFF=3, OCL_OP_MUL=4
OCL_OP_AND=9, OCL_OP_OR=10, OCL_OP_XOR=11, OCL_OP_NOT=12, OCL_OP_MIN=13, OCL_OP_MAX=14,
OCL_OP_RDIV_SCALE=15 };
// The unified entry point for every element-wise binary op (empty + UMat/OpenCL + CPU engine). Defined
// lower down (near cv::add); forward-declared here so cv::min/cv::max (above it) can use it too.
static void arithm_op(ew::TOp op, InputArray src1, InputArray src2, OutputArray dst,
InputArray mask, int dtype, int oclop, bool muldiv, const Scalar& params = Scalar(1));
// The unary counterpart (bitwise NOT, unary math later). Forward-declared for the same reason.
static void unary_op(ew::TOp op, InputArray src, OutputArray dst, InputArray mask,
int dtype, const Scalar& params = Scalar());
#ifdef HAVE_OPENCL
static const char* oclop2str[] = { "OP_ADD", "OP_SUB", "OP_RSUB", "OP_ABSDIFF",
@@ -148,309 +158,119 @@ static bool ocl_binary_op(InputArray _src1, InputArray _src2, OutputArray _dst,
#endif
static void binary_op( InputArray _src1, InputArray _src2, OutputArray _dst,
InputArray _mask, const BinaryFuncC* tab,
bool bitwise, int oclop )
// OpenCL path for the bitwise ops (and/or/xor/not), lifted out of binary_op: choose the (possibly
// scalar) operand order, size the destination, and dispatch the shared bitwise OpenCL kernel. Returns
// true if OpenCL handled the op, false to fall through to the CPU engine. NOT arrives unary
// (src2 == src1, forced-scalar). The caller gates this on a UMat operand + dims <= 2 (via CV_OCL_RUN).
#ifdef HAVE_OPENCL
static bool bitwise_op_ocl(InputArray _src1, InputArray _src2, OutputArray _dst,
InputArray _mask, int oclop)
{
const _InputArray *psrc1 = &_src1, *psrc2 = &_src2;
_InputArray::KindFlag kind1 = psrc1->kind(), kind2 = psrc2->kind();
int type1 = psrc1->type(), depth1 = CV_MAT_DEPTH(type1), cn = CV_MAT_CN(type1);
int type2 = psrc2->type(), depth2 = CV_MAT_DEPTH(type2), cn2 = CV_MAT_CN(type2);
int dims1 = psrc1->dims(), dims2 = psrc2->dims();
Size sz1 = dims1 <= 2 ? psrc1->size() : Size();
Size sz2 = dims2 <= 2 ? psrc2->size() : Size();
#ifdef HAVE_OPENCL
bool use_opencl = (kind1 == _InputArray::UMAT || kind2 == _InputArray::UMAT) &&
dims1 <= 2 && dims2 <= 2;
#endif
bool haveMask = !_mask.empty(), haveScalar = false;
BinaryFuncC func;
if( dims1 <= 2 && dims2 <= 2 && kind1 == kind2 && sz1 == sz2 && type1 == type2 && !haveMask )
{
_dst.createSameSize(*psrc1, type1);
CV_OCL_RUN(use_opencl,
ocl_binary_op(*psrc1, *psrc2, _dst, _mask, bitwise, oclop, false))
if( bitwise )
{
func = *tab;
cn = (int)CV_ELEM_SIZE(type1);
}
else
{
func = tab[depth1];
}
CV_Assert(func);
Mat src1 = psrc1->getMat(), src2 = psrc2->getMat(), dst = _dst.getMat();
Size sz = getContinuousSize2D(src1, src2, dst);
size_t len = sz.width*(size_t)cn;
if (len < INT_MAX) // FIXIT similar code below doesn't have that check
{
sz.width = (int)len;
func(src1.ptr(), src1.step, src2.ptr(), src2.step, dst.ptr(), dst.step, sz.width, sz.height, 0);
return;
}
}
int type1 = psrc1->type(), type2 = psrc2->type();
bool haveScalar = false;
if( oclop == OCL_OP_NOT )
haveScalar = true;
else if( (kind1 == _InputArray::MATX) + (kind2 == _InputArray::MATX) == 1 ||
!psrc1->sameSize(*psrc2) || type1 != type2 )
!psrc1->sameSize(*psrc2) || type1 != type2 )
{
if( checkScalar(*psrc1, type2, kind1, kind2) )
{
// src1 is a scalar; swap it with src2
swap(psrc1, psrc2);
swap(type1, type2);
swap(depth1, depth2);
swap(cn, cn2);
swap(sz1, sz2);
}
{ std::swap(psrc1, psrc2); std::swap(type1, type2); } // src1 is the scalar; swap it out
else if( !checkScalar(*psrc2, type1, kind2, kind1) )
CV_Error( cv::Error::StsUnmatchedSizes,
"The operation is neither 'array op array' (where arrays have the same size and type), "
"nor 'array op scalar', nor 'scalar op array'" );
return false;
haveScalar = true;
}
else
{
CV_Assert( psrc1->sameSize(*psrc2) && type1 == type2 );
}
size_t esz = CV_ELEM_SIZE(type1);
size_t blocksize0 = (BLOCK_SIZE + esz-1)/esz;
BinaryFunc copymask = 0;
bool reallocate = false;
if( haveMask )
{
int mtype = _mask.type();
CV_Assert( (mtype == CV_8U || mtype == CV_8S || mtype == CV_Bool) && _mask.sameSize(*psrc1));
copymask = getCopyMaskFunc(esz);
reallocate = !_dst.sameSize(*psrc1) || _dst.type() != type1;
}
AutoBuffer<uchar> _buf;
uchar *scbuf = 0, *maskbuf = 0;
_dst.createSameSize(*psrc1, type1);
// if this is mask operation and dst has been reallocated,
// we have to clear the destination
if( haveMask && reallocate )
_dst.setTo(0.);
CV_OCL_RUN(use_opencl,
ocl_binary_op(*psrc1, *psrc2, _dst, _mask, bitwise, oclop, haveScalar))
Mat src1 = psrc1->getMat(), src2 = psrc2->getMat();
Mat dst = _dst.getMat(), mask = _mask.getMat();
if( bitwise )
{
func = *tab;
cn = (int)esz;
}
else
func = tab[depth1];
CV_Assert(func);
if( !haveScalar )
{
const Mat* arrays[] = { &src1, &src2, &dst, &mask, 0 };
uchar* ptrs[4] = {};
NAryMatIterator it(arrays, ptrs);
size_t total = it.size, blocksize = total;
if( blocksize*cn > INT_MAX )
blocksize = INT_MAX/cn;
if( haveMask )
{
blocksize = std::min(blocksize, blocksize0);
_buf.allocate(blocksize*esz);
maskbuf = _buf.data();
}
for( size_t i = 0; i < it.nplanes; i++, ++it )
{
for( size_t j = 0; j < total; j += blocksize )
{
int bsz = (int)MIN(total - j, blocksize);
func( ptrs[0], 0, ptrs[1], 0, haveMask ? maskbuf : ptrs[2], 0, bsz*cn, 1, 0 );
if( haveMask )
{
copymask( maskbuf, 0, ptrs[3], 0, ptrs[2], 0, Size(bsz, 1), &esz );
ptrs[3] += bsz;
}
bsz *= (int)esz;
ptrs[0] += bsz; ptrs[1] += bsz; ptrs[2] += bsz;
}
}
}
else
{
const Mat* arrays[] = { &src1, &dst, &mask, 0 };
uchar* ptrs[3] = {};
NAryMatIterator it(arrays, ptrs);
size_t total = it.size, blocksize = std::min(total, blocksize0);
_buf.allocate(blocksize*(haveMask ? 2 : 1)*esz + 32);
scbuf = _buf.data();
maskbuf = alignPtr(scbuf + blocksize*esz, 16);
convertAndUnrollScalar( src2, src1.type(), scbuf, blocksize);
for( size_t i = 0; i < it.nplanes; i++, ++it )
{
for( size_t j = 0; j < total; j += blocksize )
{
int bsz = (int)MIN(total - j, blocksize);
func( ptrs[0], 0, scbuf, 0, haveMask ? maskbuf : ptrs[1], 0, bsz*cn, 1, 0 );
if( haveMask )
{
copymask( maskbuf, 0, ptrs[2], 0, ptrs[1], 0, Size(bsz, 1), &esz );
ptrs[2] += bsz;
}
bsz *= (int)esz;
ptrs[0] += bsz; ptrs[1] += bsz;
}
}
}
}
static BinaryFuncC* getMaxTab()
{
static BinaryFuncC maxTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)GET_OPTIMIZED(cv::hal::max8u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::max8s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::max16u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::max16s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::max32s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::max32f),
(BinaryFuncC)cv::hal::max64f,
(BinaryFuncC)cv::hal::max16f,
(BinaryFuncC)cv::hal::max16bf,
(BinaryFuncC)GET_OPTIMIZED(cv::hal::max8u), // bool
(BinaryFuncC)cv::hal::max64u,
(BinaryFuncC)cv::hal::max64s,
(BinaryFuncC)cv::hal::max32u,
0
};
return maxTab;
}
static BinaryFuncC* getMinTab()
{
static BinaryFuncC minTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)GET_OPTIMIZED(cv::hal::min8u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::min8s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::min16u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::min16s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::min32s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::min32f),
(BinaryFuncC)cv::hal::min64f,
(BinaryFuncC)cv::hal::min16f,
(BinaryFuncC)cv::hal::min16bf,
(BinaryFuncC)GET_OPTIMIZED(cv::hal::min8u), // bool
(BinaryFuncC)cv::hal::min64u,
(BinaryFuncC)cv::hal::min64s,
(BinaryFuncC)cv::hal::min32u,
0
};
return minTab;
return ocl_binary_op(*psrc1, *psrc2, _dst, _mask, true, oclop, haveScalar);
}
#endif // HAVE_OPENCL
}
// bitwise and/or/xor: OpenCL runs the shared bitwise kernel (via bitwise_op_ocl); the CPU path is the
// element-wise engine (OP_AND/OR/XOR, T x T -> T dispatched by element size). oclop = -1 tells
// arithm_op the OpenCL path was already tried, so it only runs the CPU engine. NOT is unary.
void cv::bitwise_and(InputArray a, InputArray b, OutputArray c, InputArray mask)
{
CV_INSTRUMENT_REGION();
BinaryFuncC f = (BinaryFuncC)GET_OPTIMIZED(cv::hal::and8u);
binary_op(a, b, c, mask, &f, true, OCL_OP_AND);
CV_OCL_RUN((a.isUMat() || b.isUMat() || c.isUMat()) && a.dims() <= 2 && b.dims() <= 2,
bitwise_op_ocl(a, b, c, mask, OCL_OP_AND))
arithm_op(ew::OP_AND, a, b, c, mask, -1, /*oclop=*/-1, /*muldiv=*/false);
}
void cv::bitwise_or(InputArray a, InputArray b, OutputArray c, InputArray mask)
{
CV_INSTRUMENT_REGION();
BinaryFuncC f = (BinaryFuncC)GET_OPTIMIZED(cv::hal::or8u);
binary_op(a, b, c, mask, &f, true, OCL_OP_OR);
CV_OCL_RUN((a.isUMat() || b.isUMat() || c.isUMat()) && a.dims() <= 2 && b.dims() <= 2,
bitwise_op_ocl(a, b, c, mask, OCL_OP_OR))
arithm_op(ew::OP_OR, a, b, c, mask, -1, /*oclop=*/-1, /*muldiv=*/false);
}
void cv::bitwise_xor(InputArray a, InputArray b, OutputArray c, InputArray mask)
{
CV_INSTRUMENT_REGION();
BinaryFuncC f = (BinaryFuncC)GET_OPTIMIZED(cv::hal::xor8u);
binary_op(a, b, c, mask, &f, true, OCL_OP_XOR);
CV_OCL_RUN((a.isUMat() || b.isUMat() || c.isUMat()) && a.dims() <= 2 && b.dims() <= 2,
bitwise_op_ocl(a, b, c, mask, OCL_OP_XOR))
arithm_op(ew::OP_XOR, a, b, c, mask, -1, /*oclop=*/-1, /*muldiv=*/false);
}
void cv::bitwise_not(InputArray a, OutputArray c, InputArray mask)
{
CV_INSTRUMENT_REGION();
BinaryFuncC f = (BinaryFuncC)GET_OPTIMIZED(cv::hal::not8u);
binary_op(a, a, c, mask, &f, true, OCL_OP_NOT);
CV_OCL_RUN((a.isUMat() || c.isUMat()) && a.dims() <= 2,
bitwise_op_ocl(a, a, c, mask, OCL_OP_NOT))
unary_op(ew::OP_NOT, a, c, mask, -1);
}
void cv::max( InputArray src1, InputArray src2, OutputArray dst )
{
CV_INSTRUMENT_REGION();
binary_op(src1, src2, dst, noArray(), getMaxTab(), false, OCL_OP_MAX );
arithm_op(ew::OP_MAX, src1, src2, dst, noArray(), -1, OCL_OP_MAX, false);
}
void cv::min( InputArray src1, InputArray src2, OutputArray dst )
{
CV_INSTRUMENT_REGION();
binary_op(src1, src2, dst, noArray(), getMinTab(), false, OCL_OP_MIN );
arithm_op(ew::OP_MIN, src1, src2, dst, noArray(), -1, OCL_OP_MIN, false);
}
// The concrete Mat/UMat overloads of min/max exist because C++ overload resolution needs them (a
// bare cv::min(mat, mat, mat) binds Mat& more tightly than InputArray); they must run the SAME engine
// path as the InputArray forms - NOT the legacy binary_op - so mixed types and broadcasting work.
void cv::max(const Mat& src1, const Mat& src2, Mat& dst)
{
CV_INSTRUMENT_REGION();
OutputArray _dst(dst);
binary_op(src1, src2, _dst, noArray(), getMaxTab(), false, OCL_OP_MAX );
arithm_op(ew::OP_MAX, src1, src2, dst, noArray(), -1, OCL_OP_MAX, false);
}
void cv::min(const Mat& src1, const Mat& src2, Mat& dst)
{
CV_INSTRUMENT_REGION();
OutputArray _dst(dst);
binary_op(src1, src2, _dst, noArray(), getMinTab(), false, OCL_OP_MIN );
arithm_op(ew::OP_MIN, src1, src2, dst, noArray(), -1, OCL_OP_MIN, false);
}
void cv::max(const UMat& src1, const UMat& src2, UMat& dst)
{
CV_INSTRUMENT_REGION();
OutputArray _dst(dst);
binary_op(src1, src2, _dst, noArray(), getMaxTab(), false, OCL_OP_MAX );
arithm_op(ew::OP_MAX, src1, src2, dst, noArray(), -1, OCL_OP_MAX, false);
}
void cv::min(const UMat& src1, const UMat& src2, UMat& dst)
{
CV_INSTRUMENT_REGION();
OutputArray _dst(dst);
binary_op(src1, src2, _dst, noArray(), getMinTab(), false, OCL_OP_MIN );
arithm_op(ew::OP_MIN, src1, src2, dst, noArray(), -1, OCL_OP_MIN, false);
}
@@ -461,6 +281,9 @@ void cv::min(const UMat& src1, const UMat& src2, UMat& dst)
namespace cv
{
#ifdef HAVE_OPENCL
// used by the OpenCL branch only (arithm_op_ocl working-type selection)
static int actualScalarDepth(const double* data, int len)
{
int i = 0, minval = INT_MAX, maxval = INT_MIN;
@@ -488,8 +311,6 @@ static int coerceTypes(int depth1, int depth2, bool muldiv)
((CV_ELEM_SIZE1(depth1) > 4) | (CV_ELEM_SIZE1(depth2) > 4)) != 0 ? CV_64F : CV_32F;
}
#ifdef HAVE_OPENCL
static bool ocl_arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
InputArray _mask, int wtype,
void* usrdata, int oclop,
@@ -503,6 +324,11 @@ static bool ocl_arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
if ( (haveMask || haveScalar) && cn > 4 )
return false;
#ifdef __APPLE__
if (depth1 == CV_16U && (oclop == OCL_OP_MUL || oclop == OCL_OP_MUL_SCALE))
return false;
#endif
int dtype = _dst.type(), ddepth = CV_MAT_DEPTH(dtype), wdepth = std::max(CV_32S, CV_MAT_DEPTH(wtype));
if (!doubleSupport)
wdepth = std::min(wdepth, CV_32F);
@@ -611,20 +437,20 @@ static bool ocl_arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
#endif
typedef int (*ScalarFunc)(const uchar* src, size_t step_src,
uchar* dst, size_t step_dst, int width, int height,
void* scalar, bool scalarIsFirst, int nChannels);
typedef int (*ExtendedTypeFunc)(const uchar* src1, size_t step1,
const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height,
void*);
static void arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
InputArray _mask, int dtype, BinaryFuncC* tab, bool muldiv=false,
void* usrdata=0, int oclop=-1, ExtendedTypeFunc extendedFunc = nullptr,
ScalarFunc scalarFunc = nullptr)
// OpenCL branch of the arithmetic ops, extracted from the former arithm_op (whose CPU compute is now
// the element-wise engine). Returns true if an OpenCL kernel handled the op, false to let the caller
// fall through to the CPU engine. min/max use the bitwise-style kernel; every other op the arithm
// kernel with a coerced working type. `usrdata` carries the mul/div/addWeighted scale block, `oclop`
// the kernel id. Only reached for a UMat dst; sizes/types that don't fit the scalar/same-size pattern
// (e.g. a broadcast) decline (return false) so the engine handles them on the CPU.
static bool arithm_op_ocl(InputArray _src1, InputArray _src2, OutputArray _dst,
InputArray _mask, int dtype, int oclop, bool muldiv, void* usrdata)
{
#ifndef HAVE_OPENCL
CV_UNUSED(_src1); CV_UNUSED(_src2); CV_UNUSED(_dst); CV_UNUSED(_mask);
CV_UNUSED(dtype); CV_UNUSED(oclop); CV_UNUSED(muldiv); CV_UNUSED(usrdata);
return false;
#else
const _InputArray *psrc1 = &_src1, *psrc2 = &_src2;
_InputArray::KindFlag kind1 = psrc1->kind(), kind2 = psrc2->kind();
bool haveMask = !_mask.empty();
@@ -634,61 +460,53 @@ static void arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
int wtype, dims1 = psrc1->dims(), dims2 = psrc2->dims();
Size sz1 = dims1 <= 2 ? psrc1->size() : Size();
Size sz2 = dims2 <= 2 ? psrc2->size() : Size();
#ifdef HAVE_OPENCL
bool use_opencl = OCL_PERFORMANCE_CHECK(_dst.isUMat()) && dims1 <= 2 && dims2 <= 2;
#endif
bool src1Scalar = checkScalar(*psrc1, type2, kind1, kind2);
bool src2Scalar = checkScalar(*psrc2, type1, kind2, kind1);
if( (kind1 == kind2 || cn == 1) && sz1 == sz2 && dims1 <= 2 && dims2 <= 2 && type1 == type2 &&
!haveMask && ((!_dst.fixedType() && (dtype < 0 || CV_MAT_DEPTH(dtype) == depth1)) ||
(_dst.fixedType() && _dst.type() == type1)) &&
(src1Scalar == src2Scalar) )
if (!(OCL_PERFORMANCE_CHECK(_dst.isUMat()) && dims1 <= 2 && dims2 <= 2))
return false;
// min/max: the bitwise-style OpenCL kernel (same-type element-wise, no working-type coercion).
if (oclop == OCL_OP_MIN || oclop == OCL_OP_MAX)
{
_dst.createSameSize(*psrc1, type1);
CV_OCL_RUN(use_opencl,
ocl_arithm_op(*psrc1, *psrc2, _dst, _mask,
(!usrdata ? type1 : std::max(depth1, CV_32F)),
usrdata, oclop, false))
Mat src1 = psrc1->getMat(), src2 = psrc2->getMat(), dst = _dst.getMat();
Size sz = getContinuousSize2D(src1, src2, dst, src1.channels());
if (!extendedFunc || extendedFunc(src1.ptr(), src1.step, src2.ptr(), src2.step,
dst.ptr(), dst.step, sz.width, sz.height, usrdata) != 0)
bool haveScalar = false;
if ((kind1 == _InputArray::MATX) + (kind2 == _InputArray::MATX) == 1 ||
!psrc1->sameSize(*psrc2) || type1 != type2)
{
BinaryFuncC func = tab[depth1];
CV_Assert(func);
func(src1.ptr(), src1.step, src2.ptr(), src2.step, dst.ptr(), dst.step, sz.width, sz.height, usrdata);
if (checkScalar(*psrc1, type2, kind1, kind2))
{ std::swap(psrc1, psrc2); std::swap(type1, type2); }
else if (!checkScalar(*psrc2, type1, kind2, kind1))
return false;
haveScalar = true;
}
return;
_dst.createSameSize(*psrc1, type1);
return ocl_binary_op(*psrc1, *psrc2, _dst, _mask, false, oclop, haveScalar);
}
bool haveScalar = false, swapped12 = false;
// reciprocal (scale/src): the OpenCL kernel is unary on src - use src2 for both operands (the CPU
// engine received a 0-dim `1` numerator as src1, which the recip kernel would ignore). [TODO.VP: tidy]
if (oclop == OCL_OP_RECIP_SCALE)
{
psrc1 = psrc2; kind1 = kind2; type1 = type2; depth1 = depth2; cn = cn2; dims1 = dims2; sz1 = sz2;
}
if( dims1 != dims2 || sz1 != sz2 || cn != cn2 ||
// add/sub/mul/div/absdiff/addWeighted/recip: the arithm OpenCL kernel with a coerced work type.
bool src1Scalar = checkScalar(*psrc1, type2, kind1, kind2);
bool src2Scalar = checkScalar(*psrc2, type1, kind2, kind1);
bool haveScalar = false;
if (dims1 != dims2 || sz1 != sz2 || cn != cn2 ||
(kind1 == _InputArray::MATX && (sz1 == Size(1,4) || sz1 == Size(1,1))) ||
(kind2 == _InputArray::MATX && (sz2 == Size(1,4) || sz2 == Size(1,1))) )
(kind2 == _InputArray::MATX && (sz2 == Size(1,4) || sz2 == Size(1,1))))
{
if ((type1 == CV_64F && (sz1.height == 1 || sz1.height == 4)) && src1Scalar)
{
// src1 is a scalar; swap it with src2
swap(psrc1, psrc2);
swap(sz1, sz2);
swap(type1, type2);
swap(depth1, depth2);
swap(cn, cn2);
swap(dims1, dims2);
swapped12 = true;
if( oclop == OCL_OP_SUB )
oclop = OCL_OP_RSUB;
if ( oclop == OCL_OP_DIV_SCALE )
oclop = OCL_OP_RDIV_SCALE;
std::swap(psrc1, psrc2); std::swap(sz1, sz2); std::swap(type1, type2);
std::swap(depth1, depth2); std::swap(cn, cn2); std::swap(dims1, dims2);
if (oclop == OCL_OP_SUB) oclop = OCL_OP_RSUB;
if (oclop == OCL_OP_DIV_SCALE) oclop = OCL_OP_RDIV_SCALE;
}
else if( !src2Scalar )
CV_Error( cv::Error::StsUnmatchedSizes,
"The operation is neither 'array op array' "
"(where arrays have the same size and the same number of channels), "
"nor 'array op scalar', nor 'scalar op array'" );
else if (!src2Scalar)
return false; // array op array with mismatched size/cn: engine broadcasts on the CPU
haveScalar = true;
CV_Assert((type2 == CV_64F || type2 == CV_32F) && (sz2.height == 1 || sz2.height == 4));
@@ -696,20 +514,20 @@ static void arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
{
Mat sc = psrc2->getMat();
depth2 = actualScalarDepth(sc.ptr<double>(), sz2 == Size(1, 1) ? cn2 : cn);
if( depth2 == CV_64F && CV_ELEM_SIZE1(depth1) < 8 )
if (depth2 == CV_64F && CV_ELEM_SIZE1(depth1) < 8)
depth2 = CV_32F;
}
else
depth2 = CV_64F;
}
if( dtype < 0 )
if (dtype < 0)
{
if( _dst.fixedType() )
if (_dst.fixedType())
dtype = _dst.type();
else
{
if( !haveScalar && type1 != type2 )
if (!haveScalar && type1 != type2)
CV_Error(cv::Error::StsBadArg,
"When the input arrays in add/subtract/multiply/divide functions have different types, "
"the output array type must be explicitly specified");
@@ -718,17 +536,13 @@ static void arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
}
dtype = CV_MAT_DEPTH(dtype);
if( depth1 == depth2 && dtype == depth1 )
if (depth1 == depth2 && dtype == depth1)
wtype = dtype;
else if( !muldiv )
else if (!muldiv)
{
wtype = coerceTypes(depth1, depth2, false);
wtype = coerceTypes(wtype, dtype, false);
// when the result of addition should be converted to an integer type,
// and just one of the input arrays is floating-point, it makes sense to convert that input to integer type before the operation,
// instead of converting the other input to floating-point and then converting the operation result back to integers.
if( dtype < CV_32F && (depth1 < CV_32F || depth2 < CV_32F) )
if (dtype < CV_32F && (depth1 < CV_32F || depth2 < CV_32F))
wtype = CV_32S;
}
else
@@ -737,421 +551,315 @@ static void arithm_op(InputArray _src1, InputArray _src2, OutputArray _dst,
wtype = coerceTypes(wtype, dtype, true);
}
// The scaled OpenCL kernels (mul/div by a scale, reciprocal, addWeighted) compute in float; the old
// same-type fast path forced max(depth, CV_32F). coerceTypes keeps same-type integer, so bump here.
if (oclop == OCL_OP_MUL_SCALE || oclop == OCL_OP_DIV_SCALE || oclop == OCL_OP_RDIV_SCALE ||
oclop == OCL_OP_RECIP_SCALE || oclop == OCL_OP_ADDW)
wtype = std::max(wtype, (int)CV_32F);
dtype = CV_MAKETYPE(dtype, cn);
wtype = CV_MAKETYPE(wtype, cn);
if( haveMask )
if (haveMask)
{
int mtype = _mask.type();
CV_Assert( (mtype == CV_8UC1 || mtype == CV_8SC1 || mtype == CV_Bool) && _mask.sameSize(*psrc1) );
CV_Assert((mtype == CV_8UC1 || mtype == CV_8SC1 || mtype == CV_Bool) && _mask.sameSize(*psrc1));
reallocate = !_dst.sameSize(*psrc1) || _dst.type() != dtype;
}
_dst.createSameSize(*psrc1, dtype);
if( reallocate )
if (reallocate)
_dst.setTo(0.);
CV_OCL_RUN(use_opencl,
ocl_arithm_op(*psrc1, *psrc2, _dst, _mask, wtype,
usrdata, oclop, haveScalar))
return ocl_arithm_op(*psrc1, *psrc2, _dst, _mask, wtype, usrdata, oclop, haveScalar);
#endif
}
BinaryFunc cvtsrc1 = type1 == wtype ? 0 : getConvertFunc(type1, wtype);
BinaryFunc cvtsrc2 = type2 == type1 ? cvtsrc1 : type2 == wtype ? 0 : getConvertFunc(type2, wtype);
BinaryFunc cvtdst = dtype == wtype ? 0 : getConvertFunc(wtype, dtype);
// Element-wise binary op (ADD/SUB/...) on two CPU operands - each an array Mat or a MATX scalar -
// through the new broadcasting engine. Handles mixed input types (emitBinary inserts the casts) and
// broadcasting (a scalar rides as a 0-dim per-channel CONST; arrays broadcast via outputShape/exec).
// Returns false (declining) for UMat, a write-mask, or two scalar operands, so the caller falls
// through to arithm_op. Incompatible shapes make the engine throw (matching cv::add's error).
// Does the output array already have exactly this shape+type (=> create() reuses it, no realloc)?
// Works for Mat and UMat alike (shape/type queries, no data-pointer peeking).
static bool outArrayMatches(const _OutputArray& a, const MatShape& shp, int type)
{
if (a.empty() || a.type() != type)
return false;
int sz[MatShape::MAX_DIMS];
int nd = a.sizend(sz);
if (nd != (int)shp.size())
return false;
for (int i = 0; i < nd; i++)
if (sz[i] != shp[i])
return false;
return true;
}
size_t esz1 = CV_ELEM_SIZE(type1), esz2 = CV_ELEM_SIZE(type2);
size_t dsz = CV_ELEM_SIZE(dtype), wsz = CV_ELEM_SIZE(wtype);
size_t blocksize0 = (size_t)(BLOCK_SIZE + wsz-1)/wsz;
BinaryFunc copymask = getCopyMaskFunc(dsz);
Mat src1 = psrc1->getMat(), src2 = psrc2->getMat(), dst = _dst.getMat(), mask = _mask.getMat();
// Are two operands' iteration shapes (spatial dims + channels innermost, exactly how the engine
// iterates) numpy-broadcast-compatible? Powers the scalar-like-Mat compat fallback in arithm_op.
// Cheap by design: sizend copies the dim arrays inline - no Mat headers, no allocations.
static bool shapesBroadcastCompat(InputArray a, int acn, InputArray b, int bcn)
{
int asz[MatShape::MAX_DIMS + 1], bsz[MatShape::MAX_DIMS + 1];
int ad = a.sizend(asz), bd = b.sizend(bsz);
asz[ad++] = acn;
bsz[bd++] = bcn;
for (int i1 = ad - 1, i2 = bd - 1; i1 >= 0 && i2 >= 0; i1--, i2--)
if (asz[i1] != bsz[i2] && asz[i1] != 1 && bsz[i2] != 1)
return false;
return true;
}
AutoBuffer<uchar> _buf;
uchar *buf, *maskbuf = 0, *buf1 = 0, *buf2 = 0, *wbuf = 0;
size_t bufesz = (cvtsrc1 ? wsz : 0) +
(cvtsrc2 || haveScalar ? wsz : 0) +
(cvtdst ? wsz : 0) +
(haveMask ? dsz : 0);
BinaryFuncC func = tab[CV_MAT_DEPTH(wtype)];
CV_Assert(func);
// The one entry point for every element-wise binary op (add/sub/mul/div/min/max/absdiff/addWeighted/
// reciprocal). Handles empty inputs, the UMat/OpenCL branch (arithm_op_ocl) and the CPU element-wise
// engine. `oclop` selects the OpenCL kernel; `muldiv` drives the OpenCL working-type coercion + scale;
// `params` are the op scalars (params[0]=mul/div scale; {alpha,beta,gamma} for addWeighted).
// Below this many ELEMENTS (total * channels), the per-call constant overhead of the engine path
// (program build + compile + executor setup) is comparable to the actual work - take the direct
// kernel call instead. Same tier boundary as math_op's MATH_OP_SMALL for cv::exp.
enum { ARITHM_SMALL_DIRECT = 100000 };
if( !haveScalar )
static void arithm_op(ew::TOp op, InputArray src1, InputArray src2, OutputArray dst,
InputArray mask, int dtype, int oclop, bool muldiv, const Scalar& params)
{
CV_Assert(src1.empty() == src2.empty());
if (src1.empty() && src2.empty()) // empty inputs -> empty result
{
const Mat* arrays[] = { &src1, &src2, &dst, &mask, 0 };
uchar* ptrs[4] = {};
dst.release();
if (dtype >= 0)
dst.create(0, 0, dtype);
return;
}
NAryMatIterator it(arrays, ptrs);
size_t total = it.size, blocksize = total;
// UMat -> classic OpenCL kernel. It declines (false) when OpenCL can't apply (no device, dims>2, a
// broadcast, cn>4 masked, ...) - then the CPU engine below handles it (mapping the UMat via getMat).
// oclop < 0 is a sentinel: the caller (e.g. cv::compare) already ran its own OpenCL path, so this
// helper only runs the CPU engine (a UMat operand is mapped via getMat).
if (oclop >= 0 && (src1.isUMat() || src2.isUMat() || dst.isUMat()))
{
double abg[3] = { params[0], params[1], params[2] }, scale = params[0];
void* usrdata = (op == ew::OP_ADDW) ? (void*)abg : (muldiv ? (void*)&scale : nullptr);
if (arithm_op_ocl(src1, src2, dst, mask, dtype, oclop, muldiv, usrdata))
return;
}
if( haveMask || cvtsrc1 || cvtsrc2 || cvtdst )
blocksize = std::min(blocksize, blocksize0);
const bool haveMask = !mask.empty();
const int cn1 = src1.channels(), cn2 = src2.channels();
bool s1 = isScalarArg(src1, cn2), s2 = isScalarArg(src2, cn1);
if (s1 && s2)
s1 = s2 = false; // two scalars: treat both as tiny array operands (element-wise + broadcast),
// matching arithm_op (Scalar+Scalar -> 4x1, number+number -> 1x1, ...)
// A 4x1 CV_64F Mat/UMat pseudo-scalar (see isScalarArg) is hijacked only against a REAL array:
// when the partner is itself a tiny scalar-shaped array (e.g. compare(Mat 4x1, Mat 1x1) -
// issue #8999), both are honest data and ride the broadcast. A genuine MATX Scalar never demotes.
else if (s1 && src1.kind() != _InputArray::MATX && isScalarLikeMat(src2, cn1))
s1 = false;
else if (s2 && src2.kind() != _InputArray::MATX && isScalarLikeMat(src1, cn2))
s2 = false;
_buf.allocate(bufesz*blocksize + 64);
buf = _buf.data();
if( cvtsrc1 )
// Compat fallback: Java/Python/user code passes scalars as real little Mats (the classic 4x1
// CV_64F column, a 1xcn/cnx1 vector - see isScalarLikeMat). Those normally ride broadcasting
// now, so hijack one as a per-channel scalar ONLY when the shapes do NOT broadcast - just the
// calls that would otherwise throw get the 4.x scalar semantics, every valid broadcast keeps
// its numpy meaning. Cheap: two geometry probes, and the O(ndims) shape walk runs only when a
// probe hits (never for ordinary same-size or MATX-scalar calls).
if (!s1 && !s2)
{
bool like1 = isScalarLikeMat(src1, cn2), like2 = isScalarLikeMat(src2, cn1);
if ((like1 || like2) && !shapesBroadcastCompat(src1, cn1, src2, cn2))
{
buf1 = buf, buf = alignPtr(buf + blocksize*wsz, 16);
}
if( cvtsrc2 )
{
buf2 = buf, buf = alignPtr(buf + blocksize*wsz, 16);
}
wbuf = maskbuf = buf;
if( cvtdst )
{
buf = alignPtr(buf + blocksize*wsz, 16);
}
if( haveMask )
{
maskbuf = buf;
s1 = like1; // src1 preferred when both qualify, like the old checkScalar order
s2 = like2 && !like1;
}
}
for( size_t i = 0; i < it.nplanes; i++, ++it )
// Direct Mat pointers - NO header copy / refcount atomics (the InputArrays outlive this call, so
// their data stays alive). A MAT operand is used in place via getObj(); a non-MAT one (UMat, or a
// two-scalar operand handled as an array) is mapped once into a local. EXCEPTION - in-place with a
// realloc: if dst IS one of the sources, dst.create() may free that source's data mid-op, so take a
// header copy (increfs, keeping the old data alive) of the sources in that case only.
const void* dstObj = (dst.kind() == _InputArray::MAT) ? dst.getObj() : nullptr;
const bool aliased = dstObj && (dstObj == src1.getObj() || dstObj == src2.getObj());
Mat m1loc, m2loc, mloc;
auto asMat = [&](InputArray a, Mat& loc) -> const Mat* {
if (a.kind() == _InputArray::MAT && !aliased) return (const Mat*)a.getObj();
loc = a.getMat(); return &loc;
};
const Mat* pm1 = s1 ? nullptr : asMat(src1, m1loc);
const Mat* pm2 = s2 ? nullptr : asMat(src2, m2loc);
const int adepth = s1 ? pm2->depth() : pm1->depth(); // the (first) array operand's depth
// auto result depth (no explicit dtype, no fixed-type dst): add/sub/mul/div keep the first array
// operand's depth (cv::'s dtype==-1 convention), but min/max/absdiff have NO dtype argument and
// must promote mixed inputs to their common type (min(u32,f32) in u32 would drop the float). For
// same-type inputs promoteArith(da,db)==da, so classic behaviour is preserved (e.g. absdiff of two
// s16 stays s16, saturating - the wide unsigned |a-b| is computed then cast back down by emitBinary).
int autoDepth = adepth;
if (op == ew::OP_MIN || op == ew::OP_MAX || op == ew::OP_ABSDIFF)
{
const int d1 = s1 ? ew::EW_DEPTH_NONE : pm1->depth();
const int d2 = s2 ? ew::EW_DEPTH_NONE : pm2->depth();
autoDepth = ew::promoteArith(d1, d2);
}
// result depth: explicit dtype wins; else a fixed-type dst dictates it; else the auto depth above.
const int rdepth = dtype >= 0 ? CV_MAT_DEPTH(dtype)
: (dst.fixedType() ? dst.depth() : autoDepth);
// FAST PATH for the classic hot call: two same-type same-shape continuous arrays, no mask, no
// scalar, no broadcast, result depth == input depth, and the array is SMALL. Building and
// compiling the 1-instruction program plus the BroadcastOp setup costs ~40-250ns per call -
// negligible on big images, dominant at 127x61-class sizes. Call the T x T -> T kernel directly
// over the flattened elements instead (the same tier design as math_op for cv::exp). Only ops
// whose emitBinary lowering for this exact type combination IS the plain direct kernel are
// listed - compare (boundary rewrite/flags) and divide (int guards) lower to more than one
// kernel call and keep the ordinary path. addWeighted qualifies exactly when its direct fused
// T -> T kernel exists (u8..f32; the 32/64-bit ints lower to wide-compute + cast and get {}
// from getElemwiseFunc below, falling through naturally).
if (!haveMask && !s1 && !s2 &&
(op == ew::OP_ADD || op == ew::OP_SUB || op == ew::OP_MIN || op == ew::OP_MAX ||
op == ew::OP_ABSDIFF || op == ew::OP_MUL || op == ew::OP_ADDW ||
op == ew::OP_AND || op == ew::OP_OR || op == ew::OP_XOR) &&
rdepth == pm1->depth() && pm1->type() == pm2->type() && pm1->size == pm2->size &&
pm1->isContinuous() && pm2->isContinuous() &&
pm1->total()*cn1 <= (size_t)ARITHM_SMALL_DIRECT)
{
ew::TKernel k = ew::getElemwiseFunc(op, rdepth, rdepth, ew::EW_DEPTH_NONE, rdepth);
if (k.fptr)
{
for( size_t j = 0; j < total; j += blocksize )
dst.createSameSize(src1, pm1->type()); // whole-shape transfer, not piecemeal dims+sizes
Mat dloc;
Mat* pd = (dst.kind() == _InputArray::MAT) ? (Mat*)dst.getObj() : &(dloc = dst.getMat());
if (pd->isContinuous()) // a reused non-continuous dst view falls through to the engine
{
int bsz = (int)MIN(total - j, blocksize);
Size bszn(bsz*cn, 1);
const uchar *sptr1 = ptrs[0], *sptr2 = ptrs[1];
uchar* dptr = ptrs[2];
// try to perform operation with conversion in one call
// if fail, use converter functions
uchar* opconverted = haveMask ? maskbuf : dptr;
if (!extendedFunc || extendedFunc(sptr1, 1, sptr2, 1, opconverted, (!haveMask),
bszn.width, bszn.height, usrdata) != 0)
{
if( cvtsrc1 )
{
cvtsrc1( sptr1, 1, 0, 1, buf1, 1, bszn, 0 );
sptr1 = buf1;
}
if( ptrs[0] == ptrs[1] )
{
sptr2 = sptr1;
}
else if( cvtsrc2 )
{
cvtsrc2( sptr2, 1, 0, 1, buf2, 1, bszn, 0 );
sptr2 = buf2;
}
uchar* fdst = (haveMask || cvtdst) ? wbuf : dptr;
func(sptr1, 1, sptr2, 1, fdst, (!haveMask && !cvtdst), bszn.width, bszn.height, usrdata);
if (cvtdst)
{
uchar* cdst = haveMask ? maskbuf : dptr;
cvtdst(wbuf, 1, 0, 1, cdst, 1, bszn, 0);
}
opconverted = cvtdst ? maskbuf : wbuf;
}
if (haveMask)
{
copymask(opconverted, 1, ptrs[3], 1, dptr, 1, Size(bsz, 1), &dsz);
ptrs[3] += bsz;
}
ptrs[0] += bsz*esz1; ptrs[1] += bsz*esz2; ptrs[2] += bsz*dsz;
k.fptr(pm1->data, 0, 1, pm2->data, 0, 1, nullptr, 0, 0,
pd->data, 0, (int)(pm1->total()*cn1), 1, params.val, k.flags, k.userdata);
return;
}
}
}
else
{
const Mat* arrays[] = { &src1, &dst, &mask, 0 };
uchar* ptrs[3] = {};
NAryMatIterator it(arrays, ptrs);
size_t total = it.size, blocksize = std::min(total, blocksize0);
_buf.allocate(bufesz*blocksize + 64);
buf = _buf.data();
if( cvtsrc1 )
{
buf1 = buf, buf = alignPtr(buf + blocksize * wsz, 16);
}
buf2 = buf; buf = alignPtr(buf + blocksize*wsz, 16);
wbuf = maskbuf = buf;
if( cvtdst )
{
buf = alignPtr(buf + blocksize * wsz, 16);
}
if( haveMask )
{
maskbuf = buf;
}
convertAndUnrollScalar( src2, wtype, buf2, blocksize);
for( size_t i = 0; i < it.nplanes; i++, ++it )
{
for( size_t j = 0; j < total; j += blocksize )
{
int bsz = (int)MIN(total - j, blocksize);
const uchar *sptr1 = ptrs[0];
const uchar* sptr2 = buf2;
uchar* dptr = ptrs[1];
const uchar* extSptr1 = sptr1;
const uchar* extSptr2 = sptr2;
if( swapped12 )
std::swap(extSptr1, extSptr2);
// try to perform operation in 1 call, fallback to classic way if fail
uchar* opconverted = haveMask ? maskbuf : dptr;
if (!scalarFunc || src2.total() != 1 ||
scalarFunc(extSptr1, 1, opconverted, 1, bsz, 1, (void*)extSptr2, swapped12, cn) != 0)
{
// try to perform operation with conversion in one call
// if fail, use converter functions
if (!extendedFunc || extendedFunc(extSptr1, 1, extSptr2, 1, opconverted, 1,
bsz*cn, 1, usrdata) != 0)
{
if( cvtsrc1 )
{
cvtsrc1( sptr1, 1, 0, 1, buf1, 1, Size(bsz*cn, 1), 0 );
sptr1 = buf1;
}
if( swapped12 )
std::swap(sptr1, sptr2);
uchar* fdst = ( haveMask || cvtdst ) ? wbuf : dptr;
func( sptr1, 1, sptr2, 1, fdst, 1, bsz*cn, 1, usrdata );
if (cvtdst)
{
uchar* cdst = haveMask ? maskbuf : dptr;
cvtdst(wbuf, 1, 0, 1, cdst, 1, Size(bsz*cn, 1), 0);
}
opconverted = cvtdst ? maskbuf : wbuf;
}
}
if (haveMask)
{
copymask(opconverted, 1, ptrs[2], 1, dptr, 1, Size(bsz, 1), &dsz);
ptrs[2] += bsz;
}
ptrs[0] += bsz*esz1; ptrs[1] += bsz*dsz;
}
}
}
}
static BinaryFuncC* getAddTab()
{
static BinaryFuncC addTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)GET_OPTIMIZED(cv::hal::add8u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::add8s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::add16u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::add16s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::add32s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::add32f),
(BinaryFuncC)cv::hal::add64f,
(BinaryFuncC)cv::hal::add16f,
(BinaryFuncC)cv::hal::add16bf,
0,
(BinaryFuncC)cv::hal::add64u,
(BinaryFuncC)cv::hal::add64s,
(BinaryFuncC)cv::hal::add32u,
0
// operand order = (src1, src2): an array becomes an INPUT, a scalar a flexible per-channel CONST
// read straight from the caller's inline storage (getObj()), no Mat / convertTo.
ew::TExpr p;
auto addOperand = [&](InputArray src, bool isScalar, const Mat* m, int otherCn) -> int {
if (!isScalar)
return p.addInput(m->depth());
const uchar* sp; int sd;
uchar scbuf[EW_SCALAR_BUF_SIZE]; // stack room for a UMAT scalar; addConst copies the values out
const int scn0 = scalarArgElems(src, sp, sd, scbuf), cn = otherCn;
const int scn = (cn == scn0 || (cn < 4 && scn0 == 4)) ? cn
: (scn0 == 1) ? 1 : 0;
CV_Assert(scn > 0);
return p.addConst(ew::EW_DEPTH_NONE, sd, sp, scn);
};
const int a0 = addOperand(src1, s1, pm1, cn2);
const int a1 = addOperand(src2, s2, pm2, cn1);
return addTab;
}
static int addScalar32f32fWrapper(const uchar* src, size_t step_src, uchar* dst, size_t step_dst, int width, int height,
void* scalar, bool /*scalarIsFirst*/, int nChannels)
{
int res = cv_hal_addScalar32f32f((const float*)src, step_src, (float *)dst, step_dst, width, height, (const float*)scalar, nChannels);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
// Write-mask: a single-channel 1-byte array (u8/s8/bool) the size of the output spatial shape,
// added as another broadcast input. The arithmetic result lands in a temp; a final
// select(mask, r, dst) -> dst overwrites only mask!=0 positions of the (pre-existing) output.
int sMask = 0;
const Mat* pmask = nullptr;
if (haveMask)
{
CV_Assert(mask.type() == CV_8U || mask.type() == CV_8S || mask.type() == CV_Bool);
pmask = asMat(mask, mloc);
sMask = p.addInput(pmask->depth());
}
const int sOut = p.addOutput(rdepth); // output BEFORE emit -> moveToOutput drops the
const int r = p.emitBinary(op, a0, a1, rdepth, params); // result temp (mask-free => none); scale
// rides params[0] (mul/div), {a,b,g} for addW
if (haveMask)
p.addInsn(ew::OP_SELECT, sMask, r, sOut, sOut);
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation addScalar32f32f ==> " CVAUX_STR(cv_hal_addScalar32f32f)
" returned %d (0x%08x)", res, res));
}
p.moveToOutput(r, sOut);
p.compile();
const Mat* inputs[3]; int ni = 0; // inputs in program (input-index) order
if (!s1) inputs[ni++] = pm1;
if (!s2) inputs[ni++] = pm2;
if (haveMask) inputs[ni++] = pmask;
// Pre-create the output at the broadcast shape, ALWAYS via dst.create() - never hand exec a raw
// dst Mat. Routing through the _OutputArray is what enforces its contract: a FIXED_SIZE/FIXED_TYPE
// dst (in-place 'a += b' passes '(const Mat&)a') throws here if a broadcast would change its shape
// or type (Mat(1x1) += Mat(1x4)), instead of silently reallocating - exec sees only a Mat and has
// no view of the array's fixed flags. Cost over a direct exec is negligible: exec keeps its own
// fast path for a matching-shape op, and dst.create() with an unchanged shape/type is a no-op.
// With a mask, a freshly (re)allocated dst is zeroed so mask==0 reads 0 (matches arithm_op); a
// reused dst keeps its prior content there - detect reuse by shape+type BEFORE create (correct for
// a reused UMat dst too).
MatShape oshape; int ocn;
p.outputShape(inputs, oshape, ocn);
const int otype = CV_MAKETYPE(rdepth, ocn);
const bool reused = haveMask && outArrayMatches(dst, oshape, otype);
dst.create(oshape, otype);
Mat dstloc;
Mat* pdst = (dst.kind() == _InputArray::MAT) ? (Mat*)dst.getObj() : &(dstloc = dst.getMat());
if (haveMask && !reused)
pdst->setZero(); // setZero (not setTo, which caps at 4 channels)
p.exec(inputs, pdst);
}
static int addScalar16s16sWrapper(const uchar* src, size_t step_src, uchar* dst, size_t step_dst, int width, int height,
void* scalar, bool /*scalarIsFirst*/, int nChannels)
// Unary element-wise op through the engine (bitwise NOT now; unary math later). One array input, an
// optional write-mask; result depth = dtype (>=0), else a fixed-type dst's depth, else the input
// depth. Mirrors arithm_op's build/mask/exec tail with a single input and no scalar operand. OpenCL is
// run by the caller (bitwise_op_ocl), so this is the CPU engine only.
static void unary_op(ew::TOp op, InputArray src, OutputArray dst, InputArray mask,
int dtype, const Scalar& params)
{
int res = cv_hal_addScalar16s16s((const int16_t*)src, step_src, (int16_t *)dst, step_dst, width, height, (const int16_t*)scalar, nChannels);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
if (src.empty())
{
dst.release();
if (dtype >= 0)
dst.create(0, 0, dtype);
return;
}
const bool haveMask = !mask.empty();
// in-place (dst aliases src): keep a header copy so a shape/type-changing dst.create() can't free
// src's data mid-op (matches arithm_op). A fixed-size/type dst never reallocs, so this is only for
// an explicit dtype change.
const void* dstObj = (dst.kind() == _InputArray::MAT) ? dst.getObj() : nullptr;
const bool aliased = dstObj && src.kind() == _InputArray::MAT && dstObj == src.getObj();
Mat sloc, mloc;
const Mat* psrc = (src.kind() == _InputArray::MAT && !aliased) ? (const Mat*)src.getObj()
: &(sloc = src.getMat());
const int rdepth = dtype >= 0 ? CV_MAT_DEPTH(dtype)
: (dst.fixedType() ? dst.depth() : psrc->depth());
ew::TExpr p;
const int a0 = p.addInput(psrc->depth());
int sMask = 0;
const Mat* pmask = nullptr;
if (haveMask)
{
CV_Assert(mask.type() == CV_8U || mask.type() == CV_8S || mask.type() == CV_Bool);
pmask = (mask.kind() == _InputArray::MAT) ? (const Mat*)mask.getObj() : &(mloc = mask.getMat());
sMask = p.addInput(pmask->depth());
}
const int sOut = p.addOutput(rdepth);
const int r = p.emitUnary(op, a0, rdepth, params);
if (haveMask)
p.addInsn(ew::OP_SELECT, sMask, r, sOut, sOut);
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation addScalar16s16s ==> " CVAUX_STR(cv_hal_addScalar16s16s)
" returned %d (0x%08x)", res, res));
}
}
p.moveToOutput(r, sOut);
p.compile();
static ScalarFunc getAddScalarFunc(int srcType, int dstType)
{
if (srcType == CV_32F && dstType == CV_32F)
{
return addScalar32f32fWrapper;
}
else if (srcType == CV_16S && dstType == CV_16S)
{
return addScalar16s16sWrapper;
}
else
{
return nullptr;
}
}
const Mat* inputs[2]; int ni = 0;
inputs[ni++] = psrc;
if (haveMask) inputs[ni++] = pmask;
static int sub8u32fWrapper(const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height, void* )
{
int res = cv_hal_sub8u32f(src1, step1, src2, step2, (float *)dst, step, width, height);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation sub8u32f ==> " CVAUX_STR(cv_hal_sub8u32f)
" returned %d (0x%08x)", res, res));
}
}
static int sub8s32fWrapper(const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height, void* )
{
int res = cv_hal_sub8s32f((schar*)src1, step1, (schar*)src2, step2, (float *)dst, step, width, height);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation sub8s32f ==> " CVAUX_STR(cv_hal_sub8s32f)
" returned %d (0x%08x)", res, res));
}
}
static BinaryFuncC* getSubTab()
{
static BinaryFuncC subTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)GET_OPTIMIZED(cv::hal::sub8u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::sub8s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::sub16u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::sub16s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::sub32s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::sub32f),
(BinaryFuncC)cv::hal::sub64f,
(BinaryFuncC)cv::hal::sub16f,
(BinaryFuncC)cv::hal::sub16bf,
0,
(BinaryFuncC)cv::hal::sub64u,
(BinaryFuncC)cv::hal::sub64s,
(BinaryFuncC)cv::hal::sub32u,
0
};
return subTab;
}
static ExtendedTypeFunc getSubExtFunc(int src1Type, int src2Type, int dstType)
{
if (src1Type == CV_8U && src2Type == CV_8U && dstType == CV_32F)
{
return sub8u32fWrapper;
}
else if (src1Type == CV_8S && src2Type == CV_8S && dstType == CV_32F)
{
return sub8s32fWrapper;
}
else
{
return nullptr;
}
}
static BinaryFuncC* getAbsDiffTab()
{
static BinaryFuncC absDiffTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)GET_OPTIMIZED(cv::hal::absdiff8u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::absdiff8s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::absdiff16u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::absdiff16s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::absdiff32s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::absdiff32f),
(BinaryFuncC)cv::hal::absdiff64f,
(BinaryFuncC)cv::hal::absdiff16f,
(BinaryFuncC)cv::hal::absdiff16bf,
0,
(BinaryFuncC)cv::hal::absdiff64u,
(BinaryFuncC)cv::hal::absdiff64s,
(BinaryFuncC)cv::hal::absdiff32u,
0
};
return absDiffTab;
}
static int absDiffScalar32f32fWrapper(const uchar* src, size_t step_src, uchar* dst, size_t step_dst, int width, int height,
void* scalar, bool /*scalarIsFirst*/, int nChannels)
{
int res = cv_hal_absDiffScalar32f32f((const float*)src, step_src, (float *)dst, step_dst, width, height, (const float*)scalar, nChannels);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation addScalar32f32f ==> " CVAUX_STR(cv_hal_addScalar32f32f)
" returned %d (0x%08x)", res, res));
}
}
static int absDiffScalar32s32uWrapper(const uchar* src, size_t step_src, uchar* dst, size_t step_dst, int width, int height,
void* scalar, bool /*scalarIsFirst*/, int nChannels)
{
int res = cv_hal_absDiffScalar32s32u((const int*)src, step_src, (uint32_t*)dst, step_dst, width, height, (const int*)scalar, nChannels);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation addScalar32f32f ==> " CVAUX_STR(cv_hal_addScalar32f32f)
" returned %d (0x%08x)", res, res));
}
}
static int absDiffScalar8u8uWrapper(const uchar* src, size_t step_src, uchar* dst, size_t step_dst, int width, int height,
void* scalar, bool /*scalarIsFirst*/, int nChannels)
{
int res = cv_hal_absDiffScalar8u8u((const uchar*)src, step_src, (uchar*)dst, step_dst, width, height, (const uchar*)scalar, nChannels);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation addScalar32f32f ==> " CVAUX_STR(cv_hal_addScalar32f32f)
" returned %d (0x%08x)", res, res));
}
}
static ScalarFunc getAbsDiffScalarFunc(int srcType, int dstType)
{
if (srcType == CV_32F && dstType == CV_32F)
{
return absDiffScalar32f32fWrapper;
}
// resulting type is 32U in fact
else if (srcType == CV_32S && dstType == CV_32S)
{
return absDiffScalar32s32uWrapper;
}
else if (srcType == CV_8U && dstType == CV_8U)
{
return absDiffScalar8u8uWrapper;
}
else
{
return nullptr;
}
MatShape oshape; int ocn;
p.outputShape(inputs, oshape, ocn);
const int otype = CV_MAKETYPE(rdepth, ocn);
const bool reused = haveMask && outArrayMatches(dst, oshape, otype);
dst.create(oshape, otype);
Mat dstloc;
Mat* pdst = (dst.kind() == _InputArray::MAT) ? (Mat*)dst.getObj() : &(dstloc = dst.getMat());
if (haveMask && !reused)
pdst->setZero();
p.exec(inputs, pdst);
}
}
@@ -1161,76 +869,22 @@ void cv::add( InputArray src1, InputArray src2, OutputArray dst,
{
CV_INSTRUMENT_REGION();
CV_Assert(src1.empty() == src2.empty());
if (src1.empty() && src2.empty())
{
dst.release();
if (dtype >= 0)
{
dst.create(0, 0, dtype);
}
return;
}
int sdepth = src1.depth();
if (checkScalar(src1, src1.type(), src1.kind(), _InputArray::MATX))
{
sdepth = src2.depth();
}
if (checkScalar(src2, src2.type(), src2.kind(), _InputArray::MATX))
{
sdepth = src1.depth();
}
ScalarFunc scalarFunc = getAddScalarFunc(sdepth, dtype < 0 ? dst.depth() : dtype);
arithm_op(src1, src2, dst, mask, dtype, getAddTab(), false, 0, OCL_OP_ADD, nullptr,
/* scalarFunc */ scalarFunc );
arithm_op(ew::OP_ADD, src1, src2, dst, mask, dtype, OCL_OP_ADD, false);
}
void cv::subtract( InputArray _src1, InputArray _src2, OutputArray _dst,
void cv::subtract( InputArray src1, InputArray src2, OutputArray dst,
InputArray mask, int dtype )
{
CV_INSTRUMENT_REGION();
CV_Assert(_src1.empty() == _src2.empty());
if (_src1.empty() && _src2.empty())
{
_dst.release();
if (dtype >= 0)
{
_dst.create(0, 0, dtype);
}
return;
}
ExtendedTypeFunc subExtFunc = getSubExtFunc(_src1.depth(), _src2.depth(), dtype < 0 ? _dst.depth() : dtype);
arithm_op(_src1, _src2, _dst, mask, dtype, getSubTab(), false, 0, OCL_OP_SUB,
/* extendedFunc */ subExtFunc);
arithm_op(ew::OP_SUB, src1, src2, dst, mask, dtype, OCL_OP_SUB, false);
}
void cv::absdiff( InputArray src1, InputArray src2, OutputArray dst )
{
CV_INSTRUMENT_REGION();
CV_Assert(src1.empty() == src2.empty());
if (src1.empty() && src2.empty())
{
dst.release();
return;
}
int sdepth = src1.depth();
if (checkScalar(src1, src1.type(), src1.kind(), _InputArray::MATX))
{
sdepth = src2.depth();
}
if (checkScalar(src2, src2.type(), src2.kind(), _InputArray::MATX))
{
sdepth = src1.depth();
}
ScalarFunc scalarFunc = getAbsDiffScalarFunc(sdepth, dst.depth());
arithm_op(src1, src2, dst, noArray(), -1, getAbsDiffTab(), false, 0, OCL_OP_ABSDIFF,
/* extendedFunc */ nullptr, scalarFunc);
arithm_op(ew::OP_ABSDIFF, src1, src2, dst, noArray(), -1, OCL_OP_ABSDIFF, false);
}
void cv::copyTo(InputArray _src, OutputArray _dst, InputArray _mask)
@@ -1247,102 +901,13 @@ void cv::copyTo(InputArray _src, OutputArray _dst, InputArray _mask)
namespace cv
{
static int mul8u16uWrapper(const uchar* src1, size_t step1,
const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height,
void* usrdata)
{
double scale = *((double*)usrdata);
int res = cv_hal_mul8u16u(src1, step1, src2, step2, (ushort *)dst, step, width, height, scale);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation mul8u16u ==> " CVAUX_STR(cv_hal_mul8u16u)
" returned %d (0x%08x)", res, res));
}
}
static int mul8s16sWrapper(const uchar* src1, size_t step1,
const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height,
void* usrdata)
{
double scale = *((double*)usrdata);
int res = cv_hal_mul8s16s((schar *)src1, step1, (schar *)src2, step2, (short *)dst, step, width, height, scale);
if (res == CV_HAL_ERROR_OK || res == CV_HAL_ERROR_NOT_IMPLEMENTED)
return res;
else
{
CV_Error_(cv::Error::StsInternal, ("HAL implementation mul8s16s ==> " CVAUX_STR(cv_hal_mul8s16s)
" returned %d (0x%08x)", res, res));
}
}
static BinaryFuncC* getMulTab()
{
static BinaryFuncC mulTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)cv::hal::mul8u, (BinaryFuncC)cv::hal::mul8s, (BinaryFuncC)cv::hal::mul16u,
(BinaryFuncC)cv::hal::mul16s, (BinaryFuncC)cv::hal::mul32s, (BinaryFuncC)cv::hal::mul32f,
(BinaryFuncC)cv::hal::mul64f, (BinaryFuncC)cv::hal::mul16f, (BinaryFuncC)cv::hal::mul16bf, 0,
(BinaryFuncC)cv::hal::mul64u, (BinaryFuncC)cv::hal::mul64s, (BinaryFuncC)cv::hal::mul32u, 0
};
return mulTab;
}
static ExtendedTypeFunc getMulExtFunc(int src1Type, int src2Type, int dstType)
{
if (src1Type == CV_8U && src2Type == CV_8U && dstType == CV_16U)
{
return mul8u16uWrapper;
}
else if (src1Type == CV_8S && src2Type == CV_8S && dstType == CV_16S)
{
return mul8s16sWrapper;
}
else
{
return nullptr;
}
}
static BinaryFuncC* getDivTab()
{
static BinaryFuncC divTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)cv::hal::div8u, (BinaryFuncC)cv::hal::div8s, (BinaryFuncC)cv::hal::div16u,
(BinaryFuncC)cv::hal::div16s, (BinaryFuncC)cv::hal::div32s, (BinaryFuncC)cv::hal::div32f,
(BinaryFuncC)cv::hal::div64f, (BinaryFuncC)cv::hal::div16f, (BinaryFuncC)cv::hal::div16bf, 0,
(BinaryFuncC)cv::hal::div64u, (BinaryFuncC)cv::hal::div64s, (BinaryFuncC)cv::hal::div32u, 0
};
return divTab;
}
static BinaryFuncC* getRecipTab()
{
static BinaryFuncC recipTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)cv::hal::recip8u, (BinaryFuncC)cv::hal::recip8s, (BinaryFuncC)cv::hal::recip16u,
(BinaryFuncC)cv::hal::recip16s, (BinaryFuncC)cv::hal::recip32s, (BinaryFuncC)cv::hal::recip32f,
(BinaryFuncC)cv::hal::recip64f, (BinaryFuncC)cv::hal::recip16f, (BinaryFuncC)cv::hal::recip16bf, 0,
(BinaryFuncC)cv::hal::recip64u, (BinaryFuncC)cv::hal::recip64s, (BinaryFuncC)cv::hal::recip32u, 0
};
return recipTab;
}
void multiply(InputArray src1, InputArray src2,
OutputArray dst, double scale, int dtype)
{
CV_INSTRUMENT_REGION();
ExtendedTypeFunc mulExtFunc = getMulExtFunc(src1.depth(), src2.depth(), dtype < 0 ? dst.depth() : dtype);
arithm_op(src1, src2, dst, noArray(), dtype, getMulTab(),
/* muldiv */ true, &scale, std::abs(scale - 1.0) < DBL_EPSILON ? OCL_OP_MUL : OCL_OP_MUL_SCALE,
/* extendedFunc */ mulExtFunc );
const int oclop = std::abs(scale - 1.0) < DBL_EPSILON ? OCL_OP_MUL : OCL_OP_MUL_SCALE;
arithm_op(ew::OP_MUL, src1, src2, dst, noArray(), dtype, oclop, /*muldiv*/ true, Scalar(scale));
}
void divide(InputArray src1, InputArray src2,
@@ -1350,14 +915,7 @@ void divide(InputArray src1, InputArray src2,
{
CV_INSTRUMENT_REGION();
CV_Assert(src1.empty() == src2.empty());
if (src1.empty() && src2.empty())
{
dst.release();
return;
}
arithm_op(src1, src2, dst, noArray(), dtype, getDivTab(), true, &scale, OCL_OP_DIV_SCALE);
arithm_op(ew::OP_DIV, src1, src2, dst, noArray(), dtype, OCL_OP_DIV_SCALE, /*muldiv*/ true, Scalar(scale));
}
void divide(double scale, InputArray src2,
@@ -1371,7 +929,17 @@ void divide(double scale, InputArray src2,
return;
}
arithm_op(src2, src2, dst, noArray(), dtype, getRecipTab(), true, &scale, OCL_OP_RECIP_SCALE);
// scale / src2 == scale * 1 / src2: feed a 0-dim `1` as the numerator so the CPU engine reuses the
// normal divide (scale*a/b). Give the numerator src2's OWN depth: then OP_DIV(T, T) is same-type and
// takes the fast per-type kernel (f32 work for <=16-bit / f16 / bf16, not the s32->f64 path a CV_32S
// numerator would force). An integer src2 keeps its div-by-zero->0 guard (both operands integer); a
// float src2 divides as float (1/0->inf), both matching cv::divide. dtype<0 follows src2's depth (a
// fixed-type dst still wins). The UMat path uses the dedicated reciprocal kernel (OCL_OP_RECIP_SCALE).
double one = 1;
Mat numerator(MatShape::scalar(), src2.depth(), &one);
scalarToRawData(Scalar(1), &one, src2.depth(), 1); // write `1` in src2's depth (no allocation)
const int rtype = (dtype < 0 && !dst.fixedType()) ? src2.depth() : dtype;
arithm_op(ew::OP_DIV, numerator, src2, dst, noArray(), rtype, OCL_OP_RECIP_SCALE, /*muldiv*/ true, Scalar(scale));
}
UMat UMat::mul(InputArray m, double scale) const
@@ -1385,27 +953,6 @@ UMat UMat::mul(InputArray m, double scale) const
* addWeighted *
\****************************************************************************************/
static BinaryFuncC* getAddWeightedTab()
{
static BinaryFuncC addWeightedTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)GET_OPTIMIZED(cv::hal::addWeighted8u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::addWeighted8s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::addWeighted16u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::addWeighted16s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::addWeighted32s),
(BinaryFuncC)cv::hal::addWeighted32f,
(BinaryFuncC)cv::hal::addWeighted64f,
(BinaryFuncC)cv::hal::addWeighted16f,
(BinaryFuncC)cv::hal::addWeighted16bf, 0,
(BinaryFuncC)cv::hal::addWeighted64u,
(BinaryFuncC)cv::hal::addWeighted64s,
(BinaryFuncC)cv::hal::addWeighted32u, 0
};
return addWeightedTab;
}
}
void cv::addWeighted( InputArray src1, double alpha, InputArray src2,
@@ -1413,19 +960,8 @@ void cv::addWeighted( InputArray src1, double alpha, InputArray src2,
{
CV_INSTRUMENT_REGION();
CV_Assert(src1.empty() == src2.empty());
if (src1.empty() && src2.empty())
{
dst.release();
if (dtype >= 0)
{
dst.create(0, 0, dtype);
}
return;
}
double scalars[] = {alpha, beta, gamma};
arithm_op(src1, src2, dst, noArray(), dtype, getAddWeightedTab(), true, scalars, OCL_OP_ADDW);
arithm_op(ew::OP_ADDW, src1, src2, dst, noArray(), dtype, OCL_OP_ADDW, /*muldiv*/ true,
Scalar(alpha, beta, gamma));
}
@@ -1436,28 +972,6 @@ void cv::addWeighted( InputArray src1, double alpha, InputArray src2,
namespace cv
{
static BinaryFuncC getCmpFunc(int depth)
{
static BinaryFuncC cmpTab[CV_DEPTH_MAX] =
{
(BinaryFuncC)GET_OPTIMIZED(cv::hal::cmp8u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::cmp8s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::cmp16u),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::cmp16s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::cmp32s),
(BinaryFuncC)GET_OPTIMIZED(cv::hal::cmp32f),
(BinaryFuncC)cv::hal::cmp64f,
(BinaryFuncC)cv::hal::cmp16f,
(BinaryFuncC)cv::hal::cmp16bf,
(BinaryFuncC)GET_OPTIMIZED(cv::hal::cmp8u),
(BinaryFuncC)cv::hal::cmp64u,
(BinaryFuncC)cv::hal::cmp64s,
(BinaryFuncC)cv::hal::cmp32u,
0
};
return cmpTab[depth];
}
static double getMinVal(int depth)
{
@@ -1493,6 +1007,23 @@ static bool ocl_compare(InputArray _src1, InputArray _src2, OutputArray _dst, in
if (!haveScalar && (!_src1.sameSize(_src2) || type1 != type2))
return false;
// This OpenCL kernel broadcasts a scalar's FIRST channel to all channels. The CPU path compares
// per-channel (scalar[c] for channel c), so a multichannel array against a scalar with DISTINCT
// channel values would disagree - bail to the CPU engine in that case (rare; not worth a kernel fix).
if (haveScalar && cn > 1)
{
Mat sc = _src2.getMat();
int scn = (int)sc.total();
if (scn > 1)
{
double v[4] = { 0, 0, 0, 0 };
getConvertFunc(sc.depth(), CV_64F)(sc.ptr(), 1, 0, 1, (uchar*)v, 1, Size(std::min(scn, 4), 1), 0);
for (int i = 1, n = std::min(cn, scn); i < n; i++)
if (v[i] != v[0])
return false;
}
}
int kercn = haveScalar ? cn : ocl::predictOptimalVectorWidth(_src1, _src2, _dst), rowsPerWI = dev.isIntel() ? 4 : 1;
// Workaround for bug with "?:" operator in AMD OpenCL compiler
if (depth1 >= CV_16U)
@@ -1599,6 +1130,10 @@ void cv::compare(InputArray _src1, InputArray _src2, OutputArray _dst, int op)
bool is_src1_scalar = checkScalar(_src1, _src2.type(), _src1.kind(), _src2.kind());
bool is_src2_scalar = checkScalar(_src2, _src1.type(), _src2.kind(), _src1.kind());
// exactly one scalar operand: keep it as src2 (swapping flips the ordering ops) so the OpenCL
// path sees the array-op-scalar form. Otherwise - two arrays of different size (broadcast),
// mixed types, or two scalars - the CPU engine broadcasts / promotes them, so there is NO error
// here: fall through to arithm_op (which handles scalar detection, broadcast and mixed types).
if (is_src1_scalar && !is_src2_scalar)
{
op = op == CMP_LT ? CMP_GT : op == CMP_LE ? CMP_GE :
@@ -1607,113 +1142,25 @@ void cv::compare(InputArray _src1, InputArray _src2, OutputArray _dst, int op)
compare(_src2, _src1, _dst, op);
return;
}
else if(is_src1_scalar == is_src2_scalar)
CV_Error( cv::Error::StsUnmatchedSizes,
"The operation is neither 'array op array' (where arrays have the same size and the same type), "
"nor 'array op scalar', nor 'scalar op array'" );
haveScalar = true;
haveScalar = is_src2_scalar && !is_src1_scalar;
}
CV_UNUSED(haveScalar); // consumed by CV_OCL_RUN only - a no-op without OpenCL
CV_OCL_RUN(_src1.dims() <= 2 && _src2.dims() <= 2 && OCL_PERFORMANCE_CHECK(_dst.isUMat()),
// OpenCL handles only the same-size and array-op-scalar forms; a broadcast between two arrays of
// different size falls through to the CPU engine.
CV_OCL_RUN(_src1.dims() <= 2 && _src2.dims() <= 2 && OCL_PERFORMANCE_CHECK(_dst.isUMat())
&& (haveScalar || _src1.sameSize(_src2)),
ocl_compare(_src1, _src2, _dst, op, haveScalar))
_InputArray::KindFlag kind1 = _src1.kind(), kind2 = _src2.kind();
Mat src1 = _src1.getMat(), src2 = _src2.getMat();
int depth1 = src1.depth(), depth2 = src2.depth();
if( kind1 == kind2 && src1.dims <= 2 && src2.dims <= 2 && src1.size() == src2.size() && src1.type() == src2.type() )
{
int cn = src1.channels();
_dst.createSameSize(src1, CV_8UC(cn));
Mat dst = _dst.getMat();
Size sz = getContinuousSize2D(src1, src2, dst, src1.channels());
BinaryFuncC cmpFn = getCmpFunc(depth1);
CV_Assert(cmpFn);
cmpFn(src1.ptr(), src1.step, src2.ptr(), src2.step, dst.ptr(), dst.step, sz.width, sz.height, &op);
return;
}
int cn = src1.channels();
_dst.create(src1.size, CV_8UC(cn));
src1 = src1.reshape(1); src2 = src2.reshape(1);
Mat dst = _dst.getMat().reshape(1);
size_t esz = std::max(src1.elemSize(), (size_t)1);
size_t blocksize0 = (size_t)(BLOCK_SIZE + esz-1)/esz;
BinaryFuncC func = getCmpFunc(depth1);
CV_Assert(func);
if( !haveScalar )
{
const Mat* arrays[] = { &src1, &src2, &dst, 0 };
uchar* ptrs[3] = {};
NAryMatIterator it(arrays, ptrs);
size_t total = it.size;
for( size_t i = 0; i < it.nplanes; i++, ++it )
func( ptrs[0], 0, ptrs[1], 0, ptrs[2], 0, (int)total, 1, &op );
}
else
{
const Mat* arrays[] = { &src1, &dst, 0 };
uchar* ptrs[2] = {};
NAryMatIterator it(arrays, ptrs);
size_t total = it.size, blocksize = std::min(total, blocksize0);
AutoBuffer<uchar> _buf(blocksize*esz);
uchar *buf = _buf.data();
if( ((depth1 == CV_16F) | (depth1 == CV_16BF) |
(depth1 == CV_32F) | (depth1 == CV_64F)) != 0 )
convertAndUnrollScalar( src2, depth1, buf, blocksize );
else
{
double fval=0;
BinaryFunc cvtFn = getConvertFunc(depth2, CV_64F);
CV_Assert(cvtFn);
cvtFn(src2.ptr(), 1, 0, 1, (uchar*)&fval, 1, Size(1,1), 0);
if( fval < getMinVal(depth1) )
{
dst = Scalar::all(op == CMP_GT || op == CMP_GE || op == CMP_NE ? 255 : 0);
return;
}
if( fval > getMaxVal(depth1) )
{
dst = Scalar::all(op == CMP_LT || op == CMP_LE || op == CMP_NE ? 255 : 0);
return;
}
double ival = round(fval);
if( fval != ival )
{
if( op == CMP_LT || op == CMP_GE )
ival = ceil(fval);
else if( op == CMP_LE || op == CMP_GT )
ival = floor(fval);
else
{
dst = Scalar::all(op == CMP_NE ? 255 : 0);
return;
}
}
convertAndUnrollScalar(Mat(1, 1, CV_64F, &ival), depth1, buf, blocksize);
}
for( size_t i = 0; i < it.nplanes; i++, ++it )
{
for( size_t j = 0; j < total; j += blocksize )
{
int bsz = (int)MIN(total - j, blocksize);
func( ptrs[0], 0, buf, 0, ptrs[1], 0, bsz, 1, &op);
ptrs[0] += bsz*esz;
ptrs[1] += bsz;
}
}
}
// CPU: the element-wise engine. A CMP_* code maps to the corresponding OP_CMP_* TOp; the engine
// forces a u8 mask (dtype = CV_8U) and compares in the common operand type (a fractional scalar
// threshold promotes both operands to float - matching the classic ceil/floor threshold logic).
// The scalar / broadcast / mixed-shape cases are handled inside arithm_op. oclop = -1 tells it the
// OpenCL path was already tried above (via ocl_compare), so it only runs the CPU engine.
ew::TOp cmpOp = op == CMP_EQ ? ew::OP_CMP_EQ : op == CMP_NE ? ew::OP_CMP_NE :
op == CMP_LT ? ew::OP_CMP_LT : op == CMP_LE ? ew::OP_CMP_LE :
op == CMP_GT ? ew::OP_CMP_GT : ew::OP_CMP_GE;
arithm_op(cmpOp, _src1, _src2, _dst, noArray(), CV_8U, /*oclop=*/-1, /*muldiv=*/false);
}
/****************************************************************************************\
modules/core/src/arithm.dispatch.cpp
+182 -4
View File
@@ -1,11 +1,189 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html
// of this distribution and at http://opencv.org/license.html.
// Dispatch layer for the element-wise kernels (arithm.simd.hpp). Two tiers of plain
// functions sit on top of the per-baseline get*Func_:
// - get*Func(...) : forward to the CPU-optimal kernel via CV_CPU_DISPATCH (the useful op-
// specific entry points; candidates for CV_EXPORTS later);
// - getElemwiseFunc(...): the op-level router used by the compiler.
#include "precomp.hpp"
#include "arithm_ipp.hpp"
#include "arithm_expr.hpp"
#include "arithm.simd.hpp"
#include "arithm.simd_declarations.hpp"
#define ARITHM_DISPATCHING_ONLY
#include "arithm.simd.hpp"
namespace cv { namespace ew {
// ---- tier 2: pick the kernel optimized for the current CPU ---------------------------------------
TKernel getAddFunc(int T, int R) { CV_CPU_DISPATCH(getAddFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getSubFunc(int T, int R) { CV_CPU_DISPATCH(getSubFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getMulFunc(int T, int R) { CV_CPU_DISPATCH(getMulFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getDivFunc(int T, int R, bool chk) { CV_CPU_DISPATCH(getDivFunc_, (T, R, chk), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getMinFunc(int T, int R) { CV_CPU_DISPATCH(getMinFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getMaxFunc(int T, int R) { CV_CPU_DISPATCH(getMaxFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getAbsdiffFunc(int T, int R) { CV_CPU_DISPATCH(getAbsdiffFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getHypotFunc(int T, int R) { CV_CPU_DISPATCH(getHypotFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getAtan2Func(int T, int R) { CV_CPU_DISPATCH(getAtan2Func_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getCmpFunc(TOp op, int T) { CV_CPU_DISPATCH(getCmpFunc_, (op, T), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getBitwiseFunc(TOp op, int esz) { CV_CPU_DISPATCH(getBitwiseFunc_, (op, esz), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getNotFunc(int esz) { CV_CPU_DISPATCH(getNotFunc_, (esz), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getAddWeightedFunc(int T, int R) { CV_CPU_DISPATCH(getAddWeightedFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getSelectFunc(int mdepth, int T) { CV_CPU_DISPATCH(getSelectFunc_, (mdepth, T), CV_CPU_DISPATCH_MODES_ALL); }
TKernel getClampFunc(int T) { CV_CPU_DISPATCH(getClampFunc_, (T), CV_CPU_DISPATCH_MODES_ALL); }
static TKernel getCastFunc(int sd, int dd, bool scaled)
{ CV_CPU_DISPATCH(getCastFunc_, (sd, dd, scaled),CV_CPU_DISPATCH_MODES_ALL); }
// ---- tier 3: op-level dispatcher used by the compiler --------------------------------------------
TKernel getElemwiseFunc(TOp op, int depth0, int depth1, int depth2, int rdepth)
{
(void)depth2;
if (op == OP_CAST) return getCastFunc(depth0, rdepth, false);
if (op == OP_CONVERT_SCALE) return getCastFunc(depth0, rdepth, true);
if (op == OP_ADD || op == OP_SUB)
{
if (depth0 != depth1) return {}; // operands must be the same type
return op == OP_ADD ? getAddFunc(depth0, rdepth) : getSubFunc(depth0, rdepth);
}
// OP_ADDW (addWeighted): a*alpha+b*beta+gamma; operands same type T, result R (T/f32 for small ints
// + f16/bf16/f32, f64 otherwise). alpha/beta/gamma travel in the instruction's params.
if (op == OP_ADDW)
{
if (depth0 != depth1) return {};
return getAddWeightedFunc(depth0, rdepth);
}
// OP_MUL / OP_DIV / OP_POW: operands same type T; compute in the float work type (rdepth).
if (op == OP_MUL || op == OP_DIV || op == OP_POW)
{
if (depth0 != depth1) return {};
if (op == OP_MUL) return getMulFunc(depth0, rdepth);
if (op == OP_POW) return getPowFunc(depth0, rdepth);
// integer inputs guard divide-by-zero (-> 0); floats do not (a/0 -> inf, matching cv::divide).
const bool isflt = depth0==CV_16F || depth0==CV_16BF || depth0==CV_32F || depth0==CV_64F;
return getDivFunc(depth0, rdepth, !isflt);
}
// OP_MIN / OP_MAX: T x T -> T.
if (op == OP_MIN || op == OP_MAX)
{
if (depth0 != depth1 || rdepth != depth0) return {};
return op == OP_MIN ? getMinFunc(depth0, rdepth) : getMaxFunc(depth0, rdepth);
}
// OP_ABSDIFF: result is absdiffResultDepth(T) (unsigned same width for signed).
if (op == OP_ABSDIFF)
{
if (depth0 != depth1) return {};
return getAbsdiffFunc(depth0, rdepth);
}
// OP_CMP_*: T x T -> u8 mask (0 / 1 / 255, value via TKernel::flags).
if (opCategory(op) == CAT_COMPARE)
{
if (depth0 != depth1 || rdepth != CV_8U) return {};
return getCmpFunc(op, depth0);
}
// OP_AND / OP_OR / OP_XOR: bit-pattern op, T x T -> T (same depth), dispatched by element size.
if (op == OP_AND || op == OP_OR || op == OP_XOR)
{
if (depth0 != depth1 || rdepth != depth0) return {};
return getBitwiseFunc(op, CV_ELEM_SIZE1(depth0));
}
// OP_NOT: ~x, one input, T -> T (same depth), dispatched by element size.
if (op == OP_NOT)
{
if (rdepth != depth0) return {};
return getNotFunc(CV_ELEM_SIZE1(depth0));
}
// OP_HYPOT / OP_ATAN2: T x T -> T over the float depths (integer inputs are the compiler's job).
if (op == OP_HYPOT || op == OP_ATAN2)
{
if (depth0 != depth1) return {};
return op == OP_HYPOT ? getHypotFunc(depth0, rdepth) : getAtan2Func(depth0, rdepth);
}
// unary math: T -> T over the float depths (math.simd.hpp). Anything else - integer input,
// widening/narrowing result - is the compiler's job (cast to a float working type first).
if (op == OP_SQRT || op == OP_EXP || op == OP_LOG || op == OP_SIN || op == OP_COS ||
op == OP_TANH || op == OP_ERF || op == OP_RELU)
{
if (rdepth != depth0) return {};
return getMathFunc(op, depth0);
}
// OP_SELECT: depth0 is the (1-byte, never cast) mask; both branches and dst share one depth.
if (op == OP_SELECT)
{
if (depth1 != depth2 || rdepth != depth1) return {};
return getSelectFunc(depth0, rdepth);
}
// OP_CLAMP: x, lo, hi and dst all share one depth (emitTernary unifies them).
if (op == OP_CLAMP)
{
if (depth1 != depth0 || depth2 != depth0 || rdepth != depth0) return {};
return getClampFunc(rdepth);
}
return {};
}
}} // namespace cv::ew
namespace cv { namespace hal {
// Legacy cv::hal entry point, still declared in core/hal/hal.hpp and used by external code (the
// G-API fluid backend in opencv_contrib calls it directly): forward to the element-wise engine's
// u8 multiply kernel. `scale` is a pointer to a double, as in the old contract.
void mul8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height, void* scale)
{
const double params[] = { scale ? *(const double*)scale : 1.0 };
ew::TKernel k = ew::getMulFunc(CV_8U, CV_8U);
k.fptr(src1, step1, 1, src2, step2, 1, nullptr, 0, 0, dst, step, width, height,
params, k.flags, k.userdata);
}
// Legacy cv::hal bitwise entry points, still declared in core/hal/hal.hpp and used by other modules
// (e.g. opencv_objdetect's aruco) and external code: forward to the element-wise engine's byte-wise
// bitwise kernels. No scalar params - bitwise ops ignore them.
static void bitwise8u(ew::TOp op, const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height)
{
const double noparams[4] = {};
ew::TKernel k = ew::getBitwiseFunc(op, 1);
CV_Assert(k.fptr != nullptr);
k.fptr(src1, step1, 1, src2, step2, 1, nullptr, 0, 0, dst, step, width, height,
noparams, k.flags, k.userdata);
}
void and8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height, void*)
{ bitwise8u(ew::OP_AND, src1, step1, src2, step2, dst, step, width, height); }
void or8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height, void*)
{ bitwise8u(ew::OP_OR, src1, step1, src2, step2, dst, step, width, height); }
void xor8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height, void*)
{ bitwise8u(ew::OP_XOR, src1, step1, src2, step2, dst, step, width, height); }
void not8u(const uchar* src1, size_t step1, const uchar* /*src2*/, size_t /*step2*/,
uchar* dst, size_t step, int width, int height, void*)
{
const double noparams[4] = {};
ew::TKernel k = ew::getNotFunc(1);
CV_Assert(k.fptr != nullptr);
k.fptr(src1, step1, 1, nullptr, 0, 0, nullptr, 0, 0, dst, step, width, height,
noparams, k.flags, k.userdata);
}
}} // namespace cv::hal
modules/core/src/arithm.simd.hpp
+1973 -923
View File
@@ -1,954 +1,2004 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html
// of this distribution and at http://opencv.org/license.html.
// Element-wise kernels for the new arithmetic engine, SIMD-dispatched per CPU baseline.
//
// This file is compiled once per SIMD baseline (registered via ocv_add_dispatched_file). The per-op
// entry points get*Func_(...) live in cv::ew::CV_CPU_OPTIMIZATION_NAMESPACE and return the kernel
// optimized for that baseline; the regular get*Func / getElemwiseFunc dispatchers (which forward here
// through CV_CPU_DISPATCH) live in arithm.dispatch.cpp.
//
// Kernel shape (house style of arithm.simd.hpp + convert.simd.hpp):
// - one 2D tile; per-row outer loop with stepy; dst contiguous in x.
// - stepx is restricted to {0,1}: 1 = contiguous (SIMD), 0 = broadcast-scalar along x.
// - continuity collapse 2D->1D when every operand+dst is gap-free.
// - halide right-edge backoff for the SIMD tail; scalar tail / scalar path for width<VECSZ.
// - IN-PLACE SAFE: the right-edge backoff is suppressed (scalar tail) when dst aliases an input.
//
// One template (vecBinaryKernel) covers the whole matrix: the work-vector type selects the native
// same-type saturating path vs the widening f32-hub path. Adding a binary op = add one Op functor.
#include "opencv2/core/hal/intrin.hpp"
#include "convert.hpp" // typed load/store-as helpers (cv::), and getConvert*Func
#include "arithm_expr.hpp" // the kernel contract: TOp / TKernel / KernelFunc
#include <algorithm>
#include <cmath>
#include <cstring>
//=========================================
// Declare & Define & Dispatch in one step
//=========================================
namespace cv {
// ARITHM_DISPATCHING_ONLY defined by arithm dispatch file
// Everything outside cv::ew::CV_CPU_OPTIMIZATION_NAMESPACE must be skipped in the
// declarations-only re-includes (one per dispatched mode), or it gets redefined.
#ifndef CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
#undef ARITHM_DECLARATIONS_ONLY
#ifdef CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
#define ARITHM_DECLARATIONS_ONLY
// BinaryFunc and getConvertFunc / getConvertScaleFunc come from core (precomp.hpp / convert.hpp).
#if CV_SIMD128_FP16
#undef CV_SIMD_16F
#define CV_SIMD_16F 1
#elif !defined(CV_SIMD_16F)
#define CV_SIMD_16F 0 // keep `#if CV_SIMD_16F` -Wundef-clean on builds without FP16 SIMD
#endif
#undef ARITHM_DEFINITIONS_ONLY
#if !defined(CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY) && !defined(ARITHM_DISPATCHING_ONLY)
#define ARITHM_DEFINITIONS_ONLY
#endif
///////////////////////////////////////////////////////////////////////////
namespace cv { namespace hal {
#ifndef ARITHM_DISPATCHING_ONLY
CV_CPU_OPTIMIZATION_NAMESPACE_BEGIN
#endif
#if (defined ARITHM_DECLARATIONS_ONLY) || (defined ARITHM_DEFINITIONS_ONLY)
#undef DECLARE_SIMPLE_BINARY_OP
#define DECLARE_SIMPLE_BINARY_OP(opname, type) \
void opname(const type* src1, size_t step1, const type* src2, size_t step2, \
type* dst, size_t step, int width, int height)
#undef DECLARE_SIMPLE_BINARY_OP_ALLTYPES
#define DECLARE_SIMPLE_BINARY_OP_ALLTYPES(opname) \
DECLARE_SIMPLE_BINARY_OP(opname##8u, uchar); \
DECLARE_SIMPLE_BINARY_OP(opname##8s, schar); \
DECLARE_SIMPLE_BINARY_OP(opname##16u, ushort); \
DECLARE_SIMPLE_BINARY_OP(opname##16s, short); \
DECLARE_SIMPLE_BINARY_OP(opname##32u, unsigned); \
DECLARE_SIMPLE_BINARY_OP(opname##32s, int); \
DECLARE_SIMPLE_BINARY_OP(opname##64u, uint64); \
DECLARE_SIMPLE_BINARY_OP(opname##64s, int64); \
DECLARE_SIMPLE_BINARY_OP(opname##16f, hfloat); \
DECLARE_SIMPLE_BINARY_OP(opname##16bf, bfloat); \
DECLARE_SIMPLE_BINARY_OP(opname##32f, float); \
DECLARE_SIMPLE_BINARY_OP(opname##64f, double)
DECLARE_SIMPLE_BINARY_OP_ALLTYPES(add);
DECLARE_SIMPLE_BINARY_OP_ALLTYPES(sub);
DECLARE_SIMPLE_BINARY_OP_ALLTYPES(max);
DECLARE_SIMPLE_BINARY_OP_ALLTYPES(min);
DECLARE_SIMPLE_BINARY_OP_ALLTYPES(absdiff);
void and8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2, uchar* dst, size_t step, int width, int height);
void or8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2, uchar* dst, size_t step, int width, int height);
void xor8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2, uchar* dst, size_t step, int width, int height);
void not8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2, uchar* dst, size_t step, int width, int height);
#undef DECLARE_CMP_OP
#define DECLARE_CMP_OP(opname, type) \
void opname(const type* src1, size_t step1, const type* src2, size_t step2, \
uchar* dst, size_t step, int width, int height, int cmpop)
DECLARE_CMP_OP(cmp8u, uchar);
DECLARE_CMP_OP(cmp8s, schar);
DECLARE_CMP_OP(cmp16u, ushort);
DECLARE_CMP_OP(cmp16s, short);
DECLARE_CMP_OP(cmp32u, unsigned);
DECLARE_CMP_OP(cmp32s, int);
DECLARE_CMP_OP(cmp64u, uint64);
DECLARE_CMP_OP(cmp64s, int64);
DECLARE_CMP_OP(cmp16f, hfloat);
DECLARE_CMP_OP(cmp16bf, bfloat);
DECLARE_CMP_OP(cmp32f, float);
DECLARE_CMP_OP(cmp64f, double);
#undef DECLARE_SCALED_BINARY_OP
#define DECLARE_SCALED_BINARY_OP(opname, type, scale_arg) \
void opname(const type* src1, size_t step1, const type* src2, size_t step2, \
type* dst, size_t step, int width, int height, scale_arg)
#undef DECLARE_SCALED_BINARY_OP_ALLTYPES
#define DECLARE_SCALED_BINARY_OP_ALLTYPES(opname, scale_arg) \
DECLARE_SCALED_BINARY_OP(opname##8u, uchar, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##8s, schar, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##16u, ushort, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##16s, short, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##32u, unsigned, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##32s, int, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##64u, uint64, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##64s, int64, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##16f, hfloat, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##16bf, bfloat, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##32f, float, scale_arg); \
DECLARE_SCALED_BINARY_OP(opname##64f, double, scale_arg)
DECLARE_SCALED_BINARY_OP_ALLTYPES(mul, double);
DECLARE_SCALED_BINARY_OP_ALLTYPES(div, double);
DECLARE_SCALED_BINARY_OP_ALLTYPES(recip, double);
DECLARE_SCALED_BINARY_OP_ALLTYPES(addWeighted, double weights[3]);
#endif
#ifdef ARITHM_DEFINITIONS_ONLY
#if (CV_SIMD || CV_SIMD_SCALABLE)
#define SIMD_ONLY(expr) expr
#else
#define SIMD_ONLY(expr)
#endif
static inline void vx_setall_as(const uchar* p, v_uint8& a)
{ a = vx_setall_u8(*p); }
static inline void vx_setall_as(const schar* p, v_int8& a)
{ a = vx_setall_s8(*p); }
static inline void vx_setall_as(const uchar* p, v_int16& a)
{ a = vx_setall_s16(*p); }
static inline void vx_setall_as(const schar* p, v_int16& a)
{ a = vx_setall_s16(*p); }
//=======================================
// Arithmetic and logical operations
// +, -, *, /, &, |, ^, ~, abs ...
//=======================================
static inline void vx_setall_as(const ushort* p, v_uint16& a)
{ a = vx_setall_u16(*p); }
static inline void vx_setall_as(const short* p, v_int16& a)
{ a = vx_setall_s16(*p); }
static inline void vx_setall_as(const ushort* p, v_int32& a)
{ a = vx_setall_s32(*p); }
static inline void vx_setall_as(const short* p, v_int32& a)
{ a = vx_setall_s32(*p); }
///////////////////////////// Operations //////////////////////////////////
static inline void vx_setall_as(const unsigned* p, v_uint32& a)
{ a = vx_setall_u32(*p); }
static inline void vx_setall_as(const int* p, v_int32& a)
{ a = vx_setall_s32(*p); }
#undef DEFINE_SIMPLE_BINARY_OP
#undef DEFINE_SIMPLE_BINARY_OP_F16
#undef DEFINE_SIMPLE_BINARY_OP_NOSIMD
static inline void vx_setall_as(const uchar* p, v_float32& a)
{ a = vx_setall_f32(float(*p)); }
static inline void vx_setall_as(const schar* p, v_float32& a)
{ a = vx_setall_f32(float(*p)); }
static inline void vx_setall_as(const ushort* p, v_float32& a)
{ a = vx_setall_f32(float(*p)); }
static inline void vx_setall_as(const short* p, v_float32& a)
{ a = vx_setall_f32(float(*p)); }
static inline void vx_setall_as(const float* p, v_float32& a)
{ a = vx_setall_f32(*p); }
#define DEFINE_SIMPLE_BINARY_OP(opname, T1, Tvec, scalar_op, vec_op) \
void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
T1* dst, size_t step, \
int width, int height) \
{ \
CV_INSTRUMENT_REGION(); \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
vx_store(dst + x, vec_op(vx_load(src1 + x), vx_load(src2 + x))); \
}) \
for (; x < width; x++) \
dst[x] = saturate_cast<T1>(scalar_op(src1[x], src2[x])); \
} \
SIMD_ONLY(vx_cleanup();) \
}
#define DEFINE_SIMPLE_BINARY_OP_16F(opname, T1, scalar_op, vec_op) \
void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
T1* dst, size_t step, \
int width, int height) \
{ \
CV_INSTRUMENT_REGION(); \
SIMD_ONLY(int simd_width = VTraits<v_float32>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
v_pack_store(dst + x, vec_op(vx_load_expand(src1 + x), vx_load_expand(src2 + x))); \
}) \
for (; x < width; x++) \
dst[x] = T1(scalar_op((float)src1[x], (float)src2[x])); \
} \
SIMD_ONLY(vx_cleanup();) \
}
#define DEFINE_SIMPLE_BINARY_OP_NOSIMD(opname, T1, worktype, scalar_op) \
void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
T1* dst, size_t step, \
int width, int height) \
{ \
CV_INSTRUMENT_REGION(); \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
for (int x = 0; x < width; x++) \
dst[x] = saturate_cast<T1>(scalar_op((worktype)src1[x], (worktype)src2[x])); \
} \
}
#undef scalar_add
#define scalar_add(x, y) ((x) + (y))
#undef scalar_sub
#define scalar_sub(x, y) ((x) - (y))
#undef scalar_sub_u64
#define scalar_sub_u64(x, y) ((x) <= (y) ? 0 : (x) - (y))
#undef DEFINE_SIMPLE_BINARY_OP_64F
#if (CV_SIMD_64F || CV_SIMD_SCALABLE_64F)
#define DEFINE_SIMPLE_BINARY_OP_64F(opname, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP(opname, double, v_float64, scalar_op, vec_op)
#else
#define DEFINE_SIMPLE_BINARY_OP_64F(opname, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP_NOSIMD(opname, double, double, scalar_op)
#endif
#undef DEFINE_SIMPLE_BINARY_OP_ALLTYPES
#define DEFINE_SIMPLE_BINARY_OP_ALLTYPES(opname, scalar_op, scalar_op_u64, vec_op) \
DEFINE_SIMPLE_BINARY_OP(opname##8u, uchar, v_uint8, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP(opname##8s, schar, v_int8, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP(opname##16u, ushort, v_uint16, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP(opname##16s, short, v_int16, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP_NOSIMD(opname##32u, unsigned, int64, scalar_op) \
DEFINE_SIMPLE_BINARY_OP(opname##32s, int, v_int32, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP_NOSIMD(opname##64u, uint64, uint64, scalar_op_u64) \
DEFINE_SIMPLE_BINARY_OP_NOSIMD(opname##64s, int64, int64, scalar_op) \
DEFINE_SIMPLE_BINARY_OP_16F(opname##16f, hfloat, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP_16F(opname##16bf, bfloat, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP(opname##32f, float, v_float32, scalar_op, vec_op) \
DEFINE_SIMPLE_BINARY_OP_64F(opname##64f, scalar_op, vec_op)
DEFINE_SIMPLE_BINARY_OP_ALLTYPES(add, scalar_add, scalar_add, v_add)
DEFINE_SIMPLE_BINARY_OP_ALLTYPES(sub, scalar_sub, scalar_sub_u64, v_sub)
DEFINE_SIMPLE_BINARY_OP_ALLTYPES(max, std::max, std::max, v_max)
DEFINE_SIMPLE_BINARY_OP_ALLTYPES(min, std::min, std::min, v_min)
#undef scalar_absdiff
#define scalar_absdiff(x, y) std::abs((x) - (y))
#define scalar_absdiffu(x, y) (std::max((x), (y)) - std::min((x), (y)))
DEFINE_SIMPLE_BINARY_OP(absdiff8u, uchar, v_uint8, scalar_absdiff, v_absdiff)
DEFINE_SIMPLE_BINARY_OP(absdiff8s, schar, v_int8, scalar_absdiff, v_absdiffs)
DEFINE_SIMPLE_BINARY_OP(absdiff16u, ushort, v_uint16, scalar_absdiff, v_absdiff)
DEFINE_SIMPLE_BINARY_OP(absdiff16s, short, v_int16, scalar_absdiff, v_absdiffs)
DEFINE_SIMPLE_BINARY_OP_NOSIMD(absdiff32u, unsigned, unsigned, scalar_absdiffu)
DEFINE_SIMPLE_BINARY_OP_NOSIMD(absdiff32s, int, int, scalar_absdiff)
DEFINE_SIMPLE_BINARY_OP_NOSIMD(absdiff64u, uint64, uint64, scalar_absdiffu)
DEFINE_SIMPLE_BINARY_OP_NOSIMD(absdiff64s, int64, int64, scalar_absdiff)
DEFINE_SIMPLE_BINARY_OP_16F(absdiff16f, hfloat, scalar_absdiff, v_absdiff)
DEFINE_SIMPLE_BINARY_OP_16F(absdiff16bf, bfloat, scalar_absdiff, v_absdiff)
DEFINE_SIMPLE_BINARY_OP(absdiff32f, float, v_float32, scalar_absdiff, v_absdiff)
DEFINE_SIMPLE_BINARY_OP_64F(absdiff64f, scalar_absdiff, v_absdiff)
#undef DEFINE_BINARY_LOGIC_OP
#define DEFINE_BINARY_LOGIC_OP(opname, scalar_op, vec_op) \
void opname(const uchar* src1, size_t step1, \
const uchar* src2, size_t step2, \
uchar* dst, size_t step, \
int width, int height) \
{ \
CV_INSTRUMENT_REGION(); \
int simd_width = VTraits<v_uint8>::vlanes(); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
vx_store(dst + x, vec_op(vx_load(src1 + x), vx_load(src2 + x))); \
} \
for (; x < width; x++) \
dst[x] = (uchar)(src1[x] scalar_op src2[x]); \
} \
vx_cleanup(); \
}
DEFINE_BINARY_LOGIC_OP(and8u, &, v_and)
DEFINE_BINARY_LOGIC_OP(or8u, |, v_or)
DEFINE_BINARY_LOGIC_OP(xor8u, ^, v_xor)
void not8u(const uchar* src1, size_t step1,
const uchar*, size_t,
uchar* dst, size_t step,
int width, int height)
// ---------------------------------------------------------------------------
// Saturating integer helpers the universal intrinsics lack: v_add_sat/v_sub_sat for 32-bit lanes
// (v_add/v_sub saturate 8/16-bit but WRAP 32-bit), and v_mul_sat - the FULL-precision product
// clamped to the lane type, which is exactly cv::multiply's integer semantics at scale == 1.
// Local for now - the plan is to promote these into core/hal/intrin_*.hpp as proper universal
// intrinsics for all integer types. On NEON: single-instruction add/sub (vqadd/vqsub) and the
// widening-multiply + saturating-narrow pattern (vmull + vqmovn; the vget/vcombine forms compile
// to smull2/sqxtn2 on AArch64). Elsewhere: the Hacker's Delight (2-13) bit tricks for add/sub and
// the portable v_mul_expand + saturating v_pack composition for the 8/16-bit multiplies (RVV/LSX
// also have native ones - later, in the intrinsics). 32-bit lanes have no universal widening
// multiply (no v_mul_expand for s32), so 32-bit v_mul_sat is NEON-only (EW_HAVE_MULSAT32) and
// getMulFunc_ keeps the f64 work-vector kernels on the other backends.
#if defined(__ARM_NEON)
static inline v_int32 v_add_sat(const v_int32& a, const v_int32& b) { return v_int32(vqaddq_s32(a.val, b.val)); }
static inline v_int32 v_sub_sat(const v_int32& a, const v_int32& b) { return v_int32(vqsubq_s32(a.val, b.val)); }
static inline v_uint32 v_add_sat(const v_uint32& a, const v_uint32& b) { return v_uint32(vqaddq_u32(a.val, b.val)); }
static inline v_uint32 v_sub_sat(const v_uint32& a, const v_uint32& b) { return v_uint32(vqsubq_u32(a.val, b.val)); }
#define EW_HAVE_MULSAT32 1
static inline v_uint8 v_mul_sat(const v_uint8& a, const v_uint8& b)
{
CV_INSTRUMENT_REGION();
int simd_width = VTraits<v_uint8>::vlanes();
for (; --height >= 0; src1 += step1, dst += step) {
int x = 0;
for (; x < width; x += simd_width)
{
if (x + simd_width > width) {
if (((x == 0) | (dst == src1)) != 0)
break;
x = width - simd_width;
}
vx_store(dst + x, v_not(vx_load(src1 + x)));
}
for (; x < width; x++)
dst[x] = (uchar)(~src1[x]);
uint16x8_t p0 = vmull_u8(vget_low_u8(a.val), vget_low_u8(b.val));
uint16x8_t p1 = vmull_u8(vget_high_u8(a.val), vget_high_u8(b.val));
return v_uint8(vcombine_u8(vqmovn_u16(p0), vqmovn_u16(p1)));
}
static inline v_int8 v_mul_sat(const v_int8& a, const v_int8& b)
{
int16x8_t p0 = vmull_s8(vget_low_s8(a.val), vget_low_s8(b.val));
int16x8_t p1 = vmull_s8(vget_high_s8(a.val), vget_high_s8(b.val));
return v_int8(vcombine_s8(vqmovn_s16(p0), vqmovn_s16(p1)));
}
static inline v_uint16 v_mul_sat(const v_uint16& a, const v_uint16& b)
{
uint32x4_t p0 = vmull_u16(vget_low_u16(a.val), vget_low_u16(b.val));
uint32x4_t p1 = vmull_u16(vget_high_u16(a.val), vget_high_u16(b.val));
return v_uint16(vcombine_u16(vqmovn_u32(p0), vqmovn_u32(p1)));
}
static inline v_int16 v_mul_sat(const v_int16& a, const v_int16& b)
{
int32x4_t p0 = vmull_s16(vget_low_s16(a.val), vget_low_s16(b.val));
int32x4_t p1 = vmull_s16(vget_high_s16(a.val), vget_high_s16(b.val));
return v_int16(vcombine_s16(vqmovn_s32(p0), vqmovn_s32(p1)));
}
static inline v_uint32 v_mul_sat(const v_uint32& a, const v_uint32& b)
{
uint64x2_t p0 = vmull_u32(vget_low_u32(a.val), vget_low_u32(b.val));
uint64x2_t p1 = vmull_u32(vget_high_u32(a.val), vget_high_u32(b.val));
return v_uint32(vcombine_u32(vqmovn_u64(p0), vqmovn_u64(p1)));
}
static inline v_int32 v_mul_sat(const v_int32& a, const v_int32& b)
{
int64x2_t p0 = vmull_s32(vget_low_s32(a.val), vget_low_s32(b.val));
int64x2_t p1 = vmull_s32(vget_high_s32(a.val), vget_high_s32(b.val));
return v_int32(vcombine_s32(vqmovn_s64(p0), vqmovn_s64(p1)));
}
#else
static inline v_int32 v_add_sat(const v_int32& a, const v_int32& b)
{
v_int32 res = v_add(a, b);
v_int32 ov = v_and(v_xor(a, res), v_xor(b, res)); // sign bit set iff overflow
v_int32 sat = v_xor(v_shr<31>(a), vx_setall_s32(INT_MAX)); // a >= 0 ? INT_MAX : INT_MIN
return v_select(v_lt(ov, vx_setzero_s32()), sat, res);
}
static inline v_int32 v_sub_sat(const v_int32& a, const v_int32& b)
{
v_int32 res = v_sub(a, b);
v_int32 ov = v_and(v_xor(a, b), v_xor(a, res));
v_int32 sat = v_xor(v_shr<31>(a), vx_setall_s32(INT_MAX));
return v_select(v_lt(ov, vx_setzero_s32()), sat, res);
}
static inline v_uint32 v_add_sat(const v_uint32& a, const v_uint32& b)
{
v_uint32 res = v_add(a, b);
return v_or(res, v_lt(res, a)); // wrapped => all-ones
}
static inline v_uint32 v_sub_sat(const v_uint32& a, const v_uint32& b)
{
return v_and(v_sub(a, b), v_ge(a, b)); // borrow => zero
}
static inline v_uint8 v_mul_sat(const v_uint8& a, const v_uint8& b)
{ v_uint16 p0, p1; v_mul_expand(a, b, p0, p1); return v_pack(p0, p1); }
static inline v_int8 v_mul_sat(const v_int8& a, const v_int8& b)
{ v_int16 p0, p1; v_mul_expand(a, b, p0, p1); return v_pack(p0, p1); }
static inline v_uint16 v_mul_sat(const v_uint16& a, const v_uint16& b)
{ v_uint32 p0, p1; v_mul_expand(a, b, p0, p1); return v_pack(p0, p1); }
static inline v_int16 v_mul_sat(const v_int16& a, const v_int16& b)
{ v_int32 p0, p1; v_mul_expand(a, b, p0, p1); return v_pack(p0, p1); }
#endif
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F // scalable (RVV) has v_float64 without CV_SIMD_64F
static inline void vx_setall_as(const double* p, v_float64& a) { a = vx_setall_f64(*p); }
static inline void vx_setall_as(const int* p, v_float64& a) { a = vx_setall_f64((double)*p); }
static inline void vx_setall_as(const unsigned* p, v_float64& a) { a = vx_setall_f64((double)*p); }
static inline void vx_setall_as(const int64_t* p, v_float64& a) { a = vx_setall_f64((double)*p); }
static inline void vx_setall_as(const uint64_t* p, v_float64& a) { a = vx_setall_f64((double)*p); }
#endif
static inline void vx_setall_as(const bfloat* p, v_float32& a)
{ a = vx_setall_f32(float(*p)); }
static inline void vx_setall_as(const hfloat* p, v_float32& a)
{ a = vx_setall_f32(float(*p)); }
static inline void vx_load_pair_as(const uchar* p, v_uint8& a, v_uint8& b)
{
a = vx_load(p);
b = vx_load(p + VTraits<v_uint8>::vlanes());
}
static inline void vx_load_pair_as(const schar* p, v_int8& a, v_int8& b)
{
a = vx_load(p);
b = vx_load(p + VTraits<v_int8>::vlanes());
}
static inline void vx_load_pair_as(const uchar* p, v_uint32& a, v_uint32& b)
{
v_uint16 w = vx_load_expand(p);
v_expand(w, a, b);
}
static inline void v_store_pair_as(uchar* p, const v_uint8& a, const v_uint8& b)
{
v_store(p, a);
v_store(p + VTraits<v_uint8>::vlanes(), b);
}
static inline void v_store_pair_as(schar* p, const v_int8& a, const v_int8& b)
{
v_store(p, a);
v_store(p + VTraits<v_int8>::vlanes(), b);
}
static inline void v_store_pair_as(int* p, const v_int16& a, const v_int16& b)
{
const int nlanes32 = VTraits<v_int32>::vlanes();
v_int32 v0, v1, v2, v3;
v_expand(a, v0, v1);
v_expand(b, v2, v3);
v_store(p, v0);
v_store(p + nlanes32, v1);
v_store(p + nlanes32*2, v2);
v_store(p + nlanes32*3, v3);
}
static inline void v_store_pair_as(float* p, const v_int16& a, const v_int16& b)
{
const int nlanes32 = VTraits<v_float32>::vlanes();
v_int32 v0, v1, v2, v3;
v_expand(a, v0, v1);
v_expand(b, v2, v3);
v_store(p, v_cvt_f32(v0));
v_store(p + nlanes32, v_cvt_f32(v1));
v_store(p + nlanes32*2, v_cvt_f32(v2));
v_store(p + nlanes32*3, v_cvt_f32(v3));
}
static inline void v_store_pair_as(float* p, const v_int32& a, const v_int32& b)
{
v_store(p, v_cvt_f32(a));
v_store(p + VTraits<v_float32>::vlanes(), v_cvt_f32(b));
}
static inline void v_store_pair_as(hfloat* p, const v_float32& a, const v_float32& b)
{
v_pack_store(p, a);
v_pack_store(p + VTraits<v_float32>::vlanes(), b);
}
static inline void v_store_pair_as(bfloat* p, const v_float32& a, const v_float32& b)
{
v_pack_store(p, a);
v_pack_store(p + VTraits<v_float32>::vlanes(), b);
}
// Reinterpret a comparison result (whose lane type matches the operands) as the UNSIGNED integer
// vector of the same width: u8->u8, s8->u8, u16/s16->u16, u32/s32/f32->u32. The destination type
// drives the overload, so the compare kernel needs no per-width if constexpr.
static inline void v_reinterpret_as(const v_uint8& s, v_uint8& d) { d = s; }
static inline void v_reinterpret_as(const v_int8& s, v_uint8& d) { d = v_reinterpret_as_u8(s); }
static inline void v_reinterpret_as(const v_uint16& s, v_uint16& d) { d = s; }
static inline void v_reinterpret_as(const v_int16& s, v_uint16& d) { d = v_reinterpret_as_u16(s); }
static inline void v_reinterpret_as(const v_uint32& s, v_uint32& d) { d = s; }
static inline void v_reinterpret_as(const v_int32& s, v_uint32& d) { d = v_reinterpret_as_u32(s); }
static inline void v_reinterpret_as(const v_float32& s, v_uint32& d) { d = v_reinterpret_as_u32(s); }
#if CV_SIMD_16F
static inline void v_reinterpret_as(const v_float16& s, v_uint16& d) { d = v_reinterpret_as_u16(s); }
#endif
#if CV_SIMD_64F
static inline void v_reinterpret_as(const v_float64& s, v_uint64& d) { d = v_reinterpret_as_u64(s); }
#endif
// Broadcast the compare mask value (1 or 255) across the unsigned work vector.
static inline void v_setall_mask(v_uint8& v, uchar t) { v = vx_setall_u8(t); }
static inline void v_setall_mask(v_uint16& v, uchar t) { v = vx_setall_u16(t); }
static inline void v_setall_mask(v_uint32& v, uchar t) { v = vx_setall_u32(t); }
// NOTE: v_store_pair_as(uchar*, v_uint16/v_uint32, ...) - the narrowing pack-to-u8 stores we need -
// already come from core's convert.hpp (included above), so they are not redefined here.
#if CV_SIMD_16F
static inline void vx_setall_as(const hfloat* p, v_float16& a)
{ a = vx_setall_f16(*p); }
static inline void vx_setall_as(const float* p, v_float16& a)
{ a = vx_setall_f16(hfloat(*p)); }
static inline void vx_setall_as(const uchar* p, v_float16& a)
{ a = vx_setall_f16(hfloat(float(*p))); }
static inline void vx_setall_as(const schar* p, v_float16& a)
{ a = vx_setall_f16(hfloat(float(*p))); }
static inline void vx_load_pair_as(const uchar* p, v_float16& a, v_float16& b)
{
v_uint8 v = vx_load(p);
v_uint16 v0, v1;
v_expand(v, v0, v1);
a = v_cvt_f16(v_reinterpret_as_s16(v0));
b = v_cvt_f16(v_reinterpret_as_s16(v1));
}
static inline void vx_load_pair_as(const schar* p, v_float16& a, v_float16& b)
{
v_int8 v = vx_load(p);
v_int16 v0, v1;
v_expand(v, v0, v1);
a = v_cvt_f16(v0);
b = v_cvt_f16(v1);
}
static inline void vx_load_pair_as(const hfloat* p, v_float16& a, v_float16& b)
{
a = vx_load(p);
b = vx_load(p + VTraits<v_float16>::vlanes());
}
static inline void v_store_pair_as(uchar* p, const v_float16& a, const v_float16& b)
{
v_int16 v0 = v_round(a), v1 = v_round(b);
v_store(p, v_pack_u(v0, v1));
}
static inline void v_store_pair_as(schar* p, const v_float16& a, const v_float16& b)
{
v_int16 v0 = v_round(a), v1 = v_round(b);
v_store(p, v_pack(v0, v1));
}
static inline void v_store_pair_as(hfloat* p, const v_float16& a, const v_float16& b)
{
v_store(p, a);
v_store(p + VTraits<v_float16>::vlanes(), b);
}
#endif
#endif
#endif // CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
namespace ew {
CV_CPU_OPTIMIZATION_NAMESPACE_BEGIN
// ---- per-op kernel entry points for THIS baseline (the regular dispatchers in
// arithm.dispatch.cpp reach them through CV_CPU_DISPATCH). ----
TKernel getAddFunc_(int T, int R);
TKernel getSubFunc_(int T, int R);
TKernel getMulFunc_(int T, int R);
TKernel getDivFunc_(int T, int R, bool checked);
TKernel getMinFunc_(int T, int R);
TKernel getMaxFunc_(int T, int R);
TKernel getAbsdiffFunc_(int T, int R);
TKernel getHypotFunc_(int T, int R); // hypot = sqrt(x^2+y^2), float depths, T x T -> T
TKernel getAtan2Func_(int T, int R); // atan2(y, x), radians (-pi, pi], float depths
TKernel getCmpFunc_(TOp op, int T);
TKernel getBitwiseFunc_(TOp op, int esz); // OP_AND / OP_OR / OP_XOR, by element size
TKernel getNotFunc_(int esz); // OP_NOT, by element size
TKernel getAddWeightedFunc_(int T, int R); // OP_ADDW, a*alpha+b*beta+gamma (T x T -> R)
TKernel getSelectFunc_(int mdepth, int T); // OP_SELECT: 1-byte mask, a/b/dst of T (by esz)
TKernel getClampFunc_(int T); // OP_CLAMP: min(max(x, lo), hi), all operands of T
TKernel getCastFunc_(int sdepth, int ddepth, bool scaled); // OP_CAST / OP_CONVERT_SCALE
#ifndef CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
// ===========================================================================
// Op functors (vector + scalar). New binary ops slot in here.
// ===========================================================================
struct EwAdd {
// useScalar: does the op consume the scale scalar (params[0]) in vec()/preproc()? When false,
// vecBinaryKernel's fast 2-arg branch is taken unconditionally (the check folds at compile time);
// when true, it is taken only if the runtime scale is exactly 1.
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_add(a, b); }
#if (CV_SIMD || CV_SIMD_SCALABLE)
// 32-bit lanes: v_add wraps - use the saturating version to match the scalar int64+clamp tail
static v_int32 vec(const v_int32& a, const v_int32& b) { return v_add_sat(a, b); }
static v_uint32 vec(const v_uint32& a, const v_uint32& b) { return v_add_sat(a, b); }
#endif
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
// Accumulate in the promoted type, NOT in W: for the native saturating path W is the narrow
// lane type (schar/short/...), and (W)(a+b) would wrap in 8/16 bits before saturate_cast<Tr>
// could clamp. Letting a+b promote (narrow -> int) keeps saturation for 8/16-bit outputs and
// the natural wrap for 32/64-bit (both matching cv::add). The SIMD path already saturates.
template<typename W, typename ST> static W scl(W a, W b, ST) { return W(a + b); }
};
struct EwSub {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_sub(a, b); }
#if (CV_SIMD || CV_SIMD_SCALABLE)
// 32-bit lanes: see EwAdd::vec
static v_int32 vec(const v_int32& a, const v_int32& b) { return v_sub_sat(a, b); }
static v_uint32 vec(const v_uint32& a, const v_uint32& b) { return v_sub_sat(a, b); }
#endif
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return W(a - b); } // see EwAdd::scl
// 64-bit unsigned has no wider WT to hold a-b, so the generic path would wrap on underflow.
// Saturate at 0 to match cv::subtract (8/16/32-bit already saturate via SIMD floor / wide WT).
static uint64_t scl(uint64_t a, uint64_t b, uint64_t) { return uint64_t(a >= b)*(a - b); }
};
// mul/div compute in a wide FLOAT work type (W = float for <=16-bit/f16/bf/f32, double for
// 32/64-bit/f64), matching cv::multiply/divide; the executor casts the work-type result down to
// rdepth. scalar (reference) path only for now - SIMD can follow. (No vec(): never instantiated.)
// vec(a, b, scale): the 3rd arg is the scale vector. COMMUTATIVE ops (mul, absdiff) ignore it - their
// scale is folded into the (cheaper) preproc of one operand. DIVISION is NOT commutative, so its scale
// MUST stick to the numerator; it takes the scale in vec (numerator*scale/denominator) and leaves
// preproc as identity. vecBinaryKernel always passes vscalar to vec, so every branch (incl. a broadcast
// denominator) divides correctly. The 2-arg vec (scale==1) is the both-contiguous fast path.
struct EwMul {
static constexpr bool useScalar = true;
template<typename V> static V vec(const V& a, const V& b) { return v_mul(a, b); }
#if (CV_SIMD || CV_SIMD_SCALABLE)
// integer lanes: full product clamped to the lane type (v_mul_sat) == cv::multiply semantics
// at scale 1. These drive the scale==1 fast path on FULL-width registers (Wvec1 = the native
// lane vector): whole-register loads/stores, widening happens inside the multiply itself.
// The guard matches the v_mul_sat definitions above (like EwAdd's 32-bit overloads): in
// no-SIMD builds the vector typedefs still exist (intrin_cpp), but the helpers do not.
static v_uint8 vec(const v_uint8& a, const v_uint8& b) { return v_mul_sat(a, b); }
static v_int8 vec(const v_int8& a, const v_int8& b) { return v_mul_sat(a, b); }
static v_uint16 vec(const v_uint16& a, const v_uint16& b) { return v_mul_sat(a, b); }
static v_int16 vec(const v_int16& a, const v_int16& b) { return v_mul_sat(a, b); }
#ifdef EW_HAVE_MULSAT32
static v_uint32 vec(const v_uint32& a, const v_uint32& b) { return v_mul_sat(a, b); }
static v_int32 vec(const v_int32& a, const v_int32& b) { return v_mul_sat(a, b); }
#endif
#endif
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V& s) { return v_mul(a, s); }
template<typename W, typename ST> static W scl(W a, W b, ST s) { return a * b * s; }
};
// div has two variants by the COMMON INPUT type (matching cv::'s per-type kernel choice): integer
// inputs guard divide-by-zero -> 0 (cv:: iscalar_div); float inputs do NOT guard (cv:: fscalar_div,
// a/0 -> inf), which then saturates on the cast to an integer output exactly like cv::divide.
struct EwDivInt {
static constexpr bool useScalar = true;
// integer inputs computed in the float work type: guard b==0 -> 0. Scale rides the numerator (vec).
template<typename V> static V vec(const V& a, const V& b) {
const V z = v_setzero_<V>();
return v_select(v_eq(b, z), z, v_div(a, b));
}
vx_cleanup();
}
template<typename V> static V vec(const V& a, const V& b, const V& s) {
const V z = v_setzero_<V>();
return v_select(v_eq(b, z), z, v_div(v_mul(a, s), b));
}
template<typename V> static V preproc(const V& a, const V&) { return a; } // identity: scale is in vec
template<typename W, typename ST> static W scl(W a, W b, ST s) { return b != W(0) ? a * s / b : W(0); }
};
struct EwDivFlt {
static constexpr bool useScalar = true;
template<typename V> static V vec(const V& a, const V& b) { return v_div(a, b); }
template<typename V> static V vec(const V& a, const V& b, const V& s) { return v_div(v_mul(a, s), b); }
template<typename V> static V preproc(const V& a, const V&) { return a; } // identity: scale is in vec
template<typename W, typename ST> static W scl(W a, W b, ST s) { return a * s / b; }
};
//=======================================
// Compare
//=======================================
#undef DEFINE_CMP_OP_8
#undef DEFINE_CMP_OP_16
#undef DEFINE_CMP_OP_16F
#undef DEFINE_CMP_OP_32
#undef DEFINE_CMP_OP_64
// comparison for 8-bit types
#define DEFINE_CMP_OP_8(opname, T1, Tvec, scalar_op, vec_op) \
static void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
uchar* dst, size_t step, \
int width, int height) \
{ \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == (uchar*)src1) | (dst == (uchar*)src2)) != 0) \
break; \
x = width - simd_width; \
} \
vx_store((T1*)(dst + x), vec_op(vx_load(src1 + x), vx_load(src2 + x))); \
}) \
for (; x < width; x++) \
dst[x] = (uchar)-(int)(src1[x] scalar_op src2[x]); \
} \
SIMD_ONLY(vx_cleanup();) \
}
// comparison for 16-bit integer types
#define DEFINE_CMP_OP_16(opname, T1, Tvec, scalar_op, vec_op) \
static void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
uchar* dst, size_t step, \
int width, int height) \
{ \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (x == 0) \
break; \
x = width - simd_width; \
} \
v_pack_store((schar*)(dst + x), v_reinterpret_as_s16(vec_op(vx_load(src1 + x), vx_load(src2 + x)))); \
}) \
for (; x < width; x++) \
dst[x] = (uchar)-(int)(src1[x] scalar_op src2[x]); \
} \
SIMD_ONLY(vx_cleanup();) \
}
// comparison for 16-bit floating-point types
#define DEFINE_CMP_OP_16F(opname, T1, scalar_op, vec_op) \
static void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
uchar* dst, size_t step, \
int width, int height) \
{ \
SIMD_ONLY(int simd_width = VTraits<v_float32>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width*2) \
{ \
if (x + simd_width*2 > width) { \
if (x == 0) \
break; \
x = width - simd_width*2; \
} \
auto mask0 = v_reinterpret_as_s32(vec_op(vx_load_expand(src1 + x), \
vx_load_expand(src2 + x))); \
auto mask1 = v_reinterpret_as_s32(vec_op(vx_load_expand(src1 + x + simd_width), \
vx_load_expand(src2 + x + simd_width))); \
auto mask = v_pack(mask0, mask1); \
v_pack_store((schar*)(dst + x), mask); \
}) \
for (; x < width; x++) \
dst[x] = (uchar)-(int)((float)src1[x] scalar_op (float)src2[x]); \
} \
SIMD_ONLY(vx_cleanup();) \
}
// comparison for 32-bit types
#define DEFINE_CMP_OP_32(opname, T1, Tvec, scalar_op, vec_op) \
static void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
uchar* dst, size_t step, \
int width, int height) \
{ \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width*2) \
{ \
if (x + simd_width*2 > width) { \
if (x == 0) \
break; \
x = width - simd_width*2; \
} \
auto mask0 = v_reinterpret_as_s32(vec_op(vx_load(src1 + x), \
vx_load(src2 + x))); \
auto mask1 = v_reinterpret_as_s32(vec_op(vx_load(src1 + x + simd_width), \
vx_load(src2 + x + simd_width))); \
auto mask = v_pack(mask0, mask1); \
v_pack_store((schar*)(dst + x), mask); \
}) \
for (; x < width; x++) \
dst[x] = (uchar)-(int)(src1[x] scalar_op src2[x]); \
} \
SIMD_ONLY(vx_cleanup();) \
}
// comparison for 64-bit types; don't bother with SIMD here. Hope, compiler will do it
#define DEFINE_CMP_OP_64(opname, T1, scalar_op) \
static void opname(const T1* src1, size_t step1, \
const T1* src2, size_t step2, \
uchar* dst, size_t step, \
int width, int height) \
{ \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
for (int x = 0; x < width; x++) \
dst[x] = (uchar)-(int)(src1[x] scalar_op src2[x]); \
} \
}
#undef DEFINE_CMP_OP_ALLTYPES
#define DEFINE_CMP_OP_ALLTYPES(opname, scalar_op, vec_op) \
DEFINE_CMP_OP_8(opname##8u, uchar, v_uint8, scalar_op, vec_op) \
DEFINE_CMP_OP_8(opname##8s, schar, v_int8, scalar_op, vec_op) \
DEFINE_CMP_OP_16(opname##16u, ushort, v_uint16, scalar_op, vec_op) \
DEFINE_CMP_OP_16(opname##16s, short, v_int16, scalar_op, vec_op) \
DEFINE_CMP_OP_32(opname##32u, unsigned, v_uint32, scalar_op, vec_op) \
DEFINE_CMP_OP_32(opname##32s, int, v_int32, scalar_op, vec_op) \
DEFINE_CMP_OP_64(opname##64u, uint64, scalar_op) \
DEFINE_CMP_OP_64(opname##64s, int64, scalar_op) \
DEFINE_CMP_OP_16F(opname##16f, hfloat, scalar_op, vec_op) \
DEFINE_CMP_OP_16F(opname##16bf, bfloat, scalar_op, vec_op) \
DEFINE_CMP_OP_32(opname##32f, float, v_float32, scalar_op, vec_op) \
DEFINE_CMP_OP_64(opname##64f, double, scalar_op)
DEFINE_CMP_OP_ALLTYPES(cmpeq, ==, v_eq)
DEFINE_CMP_OP_ALLTYPES(cmpne, !=, v_ne)
DEFINE_CMP_OP_ALLTYPES(cmplt, <, v_lt)
DEFINE_CMP_OP_ALLTYPES(cmple, <=, v_le)
#undef DEFINE_CMP_OP
#define DEFINE_CMP_OP(suffix, type) \
void cmp##suffix(const type* src1, size_t step1, const type* src2, size_t step2, \
uchar* dst, size_t step, int width, int height, int cmpop) \
{ \
CV_INSTRUMENT_REGION(); \
switch(cmpop) \
{ \
case CMP_LT: \
cmplt##suffix(src1, step1, src2, step2, dst, step, width, height); \
break; \
case CMP_GT: \
cmplt##suffix(src2, step2, src1, step1, dst, step, width, height); \
break; \
case CMP_LE: \
cmple##suffix(src1, step1, src2, step2, dst, step, width, height); \
break; \
case CMP_GE: \
cmple##suffix(src2, step2, src1, step1, dst, step, width, height); \
break; \
case CMP_EQ: \
cmpeq##suffix(src1, step1, src2, step2, dst, step, width, height); \
break; \
default: \
CV_Assert(cmpop == CMP_NE); \
cmpne##suffix(src1, step1, src2, step2, dst, step, width, height); \
} \
}
DEFINE_CMP_OP(8u, uchar)
DEFINE_CMP_OP(8s, schar)
DEFINE_CMP_OP(16u, ushort)
DEFINE_CMP_OP(16s, short)
DEFINE_CMP_OP(32u, unsigned)
DEFINE_CMP_OP(32s, int)
DEFINE_CMP_OP(64u, uint64)
DEFINE_CMP_OP(64s, int64)
DEFINE_CMP_OP(16f, hfloat)
DEFINE_CMP_OP(16bf, bfloat)
DEFINE_CMP_OP(32f, float)
DEFINE_CMP_OP(64f, double)
//=======================================
// Mul, Div, Recip, AddWeighted
//=======================================
#undef DEFINE_SCALED_OP_8
#undef DEFINE_SCALED_OP_16
#undef DEFINE_SCALED_OP_16F
#undef DEFINE_SCALED_OP_32
#undef DEFINE_SCALED_OP_64
#define DEFINE_SCALED_OP_8(opname, scale_arg, T1, Tvec, scalar_op, vec_op, init, pack_store_op, when_binary) \
void opname(const T1* src1, size_t step1, const T1* src2, size_t step2, \
T1* dst, size_t step, int width, int height, scale_arg) \
{ \
CV_INSTRUMENT_REGION(); \
init(); \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes()>>1;) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
v_int16 i1 = v_reinterpret_as_s16(vx_load_expand(src1 + x)); \
when_binary(v_int16 i2 = v_reinterpret_as_s16(vx_load_expand(src2 + x))); \
v_float32 f1 = v_cvt_f32(v_expand_low(i1)); \
when_binary(v_float32 f2 = v_cvt_f32(v_expand_low(i2))); \
v_float32 g1 = vec_op(); \
f1 = v_cvt_f32(v_expand_high(i1)); \
when_binary(f2 = v_cvt_f32(v_expand_high(i2))); \
v_float32 g2 = vec_op(); \
i1 = v_pack(v_round(g1), v_round(g2)); \
pack_store_op(dst + x, i1); \
}) \
for (; x < width; x++) { \
float f1 = (float)src1[x]; \
when_binary(float f2 = (float)src2[x]); \
dst[x] = saturate_cast<T1>(scalar_op()); \
} \
} \
SIMD_ONLY(vx_cleanup();) \
}
#define DEFINE_SCALED_OP_16(opname, scale_arg, T1, Tvec, scalar_op, vec_op, init, pack_store_op, when_binary) \
void opname(const T1* src1, size_t step1, const T1* src2, size_t step2, \
T1* dst, size_t step, int width, int height, scale_arg) \
{ \
CV_INSTRUMENT_REGION(); \
init() \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes()>>1;) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
v_int32 i1 = v_reinterpret_as_s32(vx_load_expand(src1 + x)); \
when_binary(v_int32 i2 = v_reinterpret_as_s32(vx_load_expand(src2 + x))); \
v_float32 f1 = v_cvt_f32(i1); \
when_binary(v_float32 f2 = v_cvt_f32(i2)); \
f1 = vec_op(); \
i1 = v_round(f1); \
pack_store_op(dst + x, i1); \
}) \
for (; x < width; x++) { \
float f1 = (float)src1[x]; \
when_binary(float f2 = (float)src2[x]); \
dst[x] = saturate_cast<T1>(scalar_op()); \
} \
} \
SIMD_ONLY(vx_cleanup();) \
}
#define DEFINE_SCALED_OP_16F(opname, scale_arg, T1, scalar_op, vec_op, init, when_binary) \
void opname(const T1* src1, size_t step1, const T1* src2, size_t step2, \
T1* dst, size_t step, int width, int height, scale_arg) \
{ \
CV_INSTRUMENT_REGION(); \
init() \
SIMD_ONLY(int simd_width = VTraits<v_float32>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
v_float32 f1 = vx_load_expand(src1 + x); \
when_binary(v_float32 f2 = vx_load_expand(src2 + x)); \
f1 = vec_op(); \
v_pack_store(dst + x, f1); \
}) \
for (; x < width; x++) { \
float f1 = (float)src1[x]; \
when_binary(float f2 = (float)src2[x]); \
dst[x] = saturate_cast<T1>(scalar_op()); \
} \
} \
SIMD_ONLY(vx_cleanup();) \
}
#define DEFINE_SCALED_OP_32(opname, scale_arg, T1, Tvec, scalar_op, vec_op, init, load_op, store_op, when_binary) \
void opname(const T1* src1, size_t step1, const T1* src2, size_t step2, \
T1* dst, size_t step, int width, int height, scale_arg) \
{ \
CV_INSTRUMENT_REGION(); \
init() \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
v_float32 f1 = load_op(src1 + x); \
when_binary(v_float32 f2 = load_op(src2 + x)); \
f1 = vec_op(); \
store_op(dst + x, f1); \
}) \
for (; x < width; x++) { \
float f1 = (float)src1[x]; \
when_binary(float f2 = (float)src2[x]); \
dst[x] = saturate_cast<T1>(scalar_op()); \
} \
} \
SIMD_ONLY(vx_cleanup();) \
}
#define DEFINE_SCALED_OP_64F_(opname, scale_arg, T1, Tvec, scalar_op, vec_op, init, when_binary) \
void opname(const T1* src1, size_t step1, const T1* src2, size_t step2, \
T1* dst, size_t step, int width, int height, scale_arg) \
{ \
CV_INSTRUMENT_REGION(); \
init() \
SIMD_ONLY(int simd_width = VTraits<Tvec>::vlanes();) \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
SIMD_ONLY(for (; x < width; x += simd_width) \
{ \
if (x + simd_width > width) { \
if (((x == 0) | (dst == src1) | (dst == src2)) != 0) \
break; \
x = width - simd_width; \
} \
v_float64 f1 = vx_load(src1 + x); \
when_binary(v_float64 f2 = vx_load(src2 + x)); \
f1 = vec_op(); \
v_store(dst + x, f1); \
}) \
for (; x < width; x++) { \
double f1 = (double)src1[x]; \
when_binary(double f2 = (double)src2[x]); \
dst[x] = saturate_cast<T1>(scalar_op()); \
} \
} \
SIMD_ONLY(vx_cleanup();) \
}
#define DEFINE_SCALED_OP_NOSIMD(opname, scale_arg, T1, worktype, scalar_op, init, when_binary) \
void opname(const T1* src1, size_t step1, const T1* src2, size_t step2, \
T1* dst, size_t step, int width, int height, scale_arg) \
{ \
CV_INSTRUMENT_REGION(); \
init() \
step1 /= sizeof(T1); \
step2 /= sizeof(T1); \
step /= sizeof(T1); \
for (; --height >= 0; src1 += step1, src2 += step2, dst += step) { \
int x = 0; \
for (; x < width; x++) { \
worktype f1 = (worktype)src1[x]; \
when_binary(worktype f2 = (worktype)src2[x]); \
dst[x] = saturate_cast<T1>(scalar_op()); \
} \
} \
}
#define init_muldiv_f32() \
float sscale = (float)scale; \
SIMD_ONLY(v_float32 vzero = vx_setzero_f32(); \
v_float32 vscale = v_add(vx_setall_f32(sscale), vzero);)
#define init_addw_f32() \
float sw1 = (float)weights[0]; \
float sw2 = (float)weights[1]; \
float sdelta = (float)weights[2];\
SIMD_ONLY(v_float32 vw1 = vx_setall_f32(sw1); \
v_float32 vw2 = vx_setall_f32(sw2); \
v_float32 vdelta = vx_setall_f32(sdelta);)
#undef init_muldiv_nosimd_f32
#define init_muldiv_nosimd_f32() \
float sscale = (float)scale;
#undef init_addw_nosimd_f32
#define init_addw_nosimd_f32() \
float sw1 = (float)weights[0]; \
float sw2 = (float)weights[1]; \
float sdelta = (float)weights[2];
#undef init_muldiv_nosimd_f64
#undef init_addw_nosimd_f64
#define init_muldiv_nosimd_f64() \
double sscale = scale;
#define init_addw_nosimd_f64() \
double sw1 = weights[0]; \
double sw2 = weights[1]; \
double sdelta = weights[2];
#if (CV_SIMD_64F || CV_SIMD_SCALABLE_64F)
#define DEFINE_SCALED_OP_64F(opname, scale_arg, scalar_op, vec_op, init, when_binary) \
DEFINE_SCALED_OP_64F_(opname, scale_arg, double, v_float64, scalar_op, vec_op, init, when_binary)
#define init_muldiv_f64() \
double sscale = (double)scale; \
SIMD_ONLY(v_float64 vzero = vx_setzero_f64(); \
v_float64 vscale = v_add(vx_setall_f64(sscale), vzero);)
#define init_addw_f64() \
double sw1 = weights[0]; \
double sw2 = weights[1]; \
double sdelta = weights[2];\
SIMD_ONLY(v_float64 vw1 = vx_setall_f64(sw1); \
v_float64 vw2 = vx_setall_f64(sw2); \
v_float64 vdelta = vx_setall_f64(sdelta);)
#else
#define DEFINE_SCALED_OP_64F(opname, scale_arg, scalar_op, vec_op, init, when_binary) \
DEFINE_SCALED_OP_NOSIMD(opname, scale_arg, double, double, scalar_op, init, when_binary)
#define init_muldiv_f64() init_muldiv_nosimd_f64()
#define init_addw_f64() init_addw_nosimd_f64()
#endif
#undef scalar_mul
#undef vec_mul
#undef iscalar_div
#undef ivec_div
#undef fscalar_div
#undef fvec_div
#undef scalar_addw
#undef vec_addw
#define scalar_mul() ((f1)*(f2)*sscale)
#define vec_mul() v_mul(v_mul((f1), vscale), (f2))
#define iscalar_div() ((f2)!=0? (f1)*sscale/(f2) : 0)
#define ivec_div() v_select(v_eq((f2), vzero), vzero, v_div(v_mul((f1), vscale), (f2)))
#define fscalar_div() ((f1)*sscale/(f2))
#define fvec_div() v_div(v_mul((f1), vscale), (f2))
#define iscalar_recip() ((f1)!=0? sscale/(f1) : 0)
#define ivec_recip() v_select(v_eq((f1), vzero), vzero, v_div(vscale, (f1)))
#define fscalar_recip() (sscale/(f1))
#define fvec_recip() v_div(vscale, (f1))
#define scalar_addw() ((f1)*sw1 + (f2)*sw2 + sdelta)
#define vec_addw() v_fma((f1), vw1, v_fma((f2), vw2, vdelta))
#undef load_as_f32
#undef store_as_s32
#define load_as_f32(addr) v_cvt_f32(vx_load(addr))
#define store_as_s32(addr, x) v_store((addr), v_round(x))
#undef this_is_binary
#undef this_is_unary
#define this_is_binary(expr) expr
#define this_is_unary(expr)
#undef DEFINE_SCALED_OP_ALLTYPES
#define DEFINE_SCALED_OP_ALLTYPES(opname, scale_arg, iscalar_op, fscalar_op, ivec_op, fvec_op, init, when_binary) \
DEFINE_SCALED_OP_8(opname##8u, scale_arg, uchar, v_uint8, iscalar_op, ivec_op, init##_f32, v_pack_u_store, when_binary) \
DEFINE_SCALED_OP_8(opname##8s, scale_arg, schar, v_int8, iscalar_op, ivec_op, init##_f32, v_pack_store, when_binary) \
DEFINE_SCALED_OP_16(opname##16u, scale_arg, ushort, v_uint16, iscalar_op, ivec_op, init##_f32, v_pack_u_store, when_binary) \
DEFINE_SCALED_OP_16(opname##16s, scale_arg, short, v_int16, iscalar_op, ivec_op, init##_f32, v_pack_store, when_binary) \
DEFINE_SCALED_OP_NOSIMD(opname##32u, scale_arg, unsigned, double, iscalar_op, init##_nosimd_f64, when_binary) \
DEFINE_SCALED_OP_NOSIMD(opname##32s, scale_arg, int, double, iscalar_op, init##_nosimd_f64, when_binary) \
DEFINE_SCALED_OP_NOSIMD(opname##64u, scale_arg, uint64, double, iscalar_op, init##_nosimd_f64, when_binary) \
DEFINE_SCALED_OP_NOSIMD(opname##64s, scale_arg, int64, double, iscalar_op, init##_nosimd_f64, when_binary) \
DEFINE_SCALED_OP_32(opname##32f, scale_arg, float, v_float32, fscalar_op, fvec_op, init##_f32, vx_load, v_store, when_binary) \
DEFINE_SCALED_OP_64F(opname##64f, scale_arg, fscalar_op, fvec_op, init##_f64, when_binary) \
DEFINE_SCALED_OP_16F(opname##16f, scale_arg, hfloat, fscalar_op, fvec_op, init##_f32, when_binary) \
DEFINE_SCALED_OP_16F(opname##16bf, scale_arg, bfloat, fscalar_op, fvec_op, init##_f32, when_binary)
DEFINE_SCALED_OP_ALLTYPES(mul, double scale, scalar_mul, scalar_mul, vec_mul, vec_mul, init_muldiv, this_is_binary)
DEFINE_SCALED_OP_ALLTYPES(div, double scale, iscalar_div, fscalar_div, ivec_div, fvec_div, init_muldiv, this_is_binary)
DEFINE_SCALED_OP_ALLTYPES(addWeighted, double weights[3], scalar_addw, scalar_addw, vec_addw, vec_addw, init_addw, this_is_binary)
DEFINE_SCALED_OP_ALLTYPES(recip, double scale, iscalar_recip, fscalar_recip, ivec_recip, fvec_recip, init_muldiv, this_is_unary)
#endif
#ifdef ARITHM_DISPATCHING_ONLY
#undef DEFINE_BINARY_OP_DISPATCHER
#define DEFINE_BINARY_OP_DISPATCHER(opname, decl_type, type) \
void opname(const decl_type* src1, size_t step1, const decl_type* src2, size_t step2, \
decl_type* dst, size_t step, int width, int height, void*) \
{ \
CV_INSTRUMENT_REGION(); \
CALL_HAL(opname, cv_hal_##opname, src1, step1, src2, step2, dst, step, width, height) \
CV_CPU_DISPATCH(opname, ((const type*)src1, step1, (const type*)src2, step2, \
(type*)dst, step, width, height), CV_CPU_DISPATCH_MODES_ALL); \
}
#define DEFINE_BINARY_OP_DISPATCHER_ALLTYPES(opname) \
DEFINE_BINARY_OP_DISPATCHER(opname##8u, uchar, uchar) \
DEFINE_BINARY_OP_DISPATCHER(opname##8s, schar, schar) \
DEFINE_BINARY_OP_DISPATCHER(opname##16u, ushort, ushort) \
DEFINE_BINARY_OP_DISPATCHER(opname##16s, short, short) \
DEFINE_BINARY_OP_DISPATCHER(opname##32u, unsigned, unsigned) \
DEFINE_BINARY_OP_DISPATCHER(opname##32s, int, int) \
DEFINE_BINARY_OP_DISPATCHER(opname##64u, uint64, uint64) \
DEFINE_BINARY_OP_DISPATCHER(opname##64s, int64, int64) \
DEFINE_BINARY_OP_DISPATCHER(opname##16f, cv_hal_f16, hfloat) \
DEFINE_BINARY_OP_DISPATCHER(opname##16bf, cv_hal_bf16, bfloat) \
DEFINE_BINARY_OP_DISPATCHER(opname##32f, float, float) \
DEFINE_BINARY_OP_DISPATCHER(opname##64f, double, double)
DEFINE_BINARY_OP_DISPATCHER_ALLTYPES(add)
DEFINE_BINARY_OP_DISPATCHER_ALLTYPES(sub)
DEFINE_BINARY_OP_DISPATCHER_ALLTYPES(max)
DEFINE_BINARY_OP_DISPATCHER_ALLTYPES(min)
DEFINE_BINARY_OP_DISPATCHER_ALLTYPES(absdiff)
DEFINE_BINARY_OP_DISPATCHER(and8u, uchar, uchar)
DEFINE_BINARY_OP_DISPATCHER(or8u, uchar, uchar)
DEFINE_BINARY_OP_DISPATCHER(xor8u, uchar, uchar)
void not8u(const uchar* src1, size_t step1, const uchar* src2, size_t step2,
uchar* dst, size_t step, int width, int height, void*)
// atan2(y, x) in RADIANS over the standard C range (-pi, pi] - the fastAtan2 minimax polynomial
// (mathfuncs_core.simd.hpp v_atan_f32) with the 180/pi factor dropped and the C quadrant logic
// (fastAtan2 returns degrees in [0, 360)). Absolute accuracy ~1e-5 rad, same as cv::fastAtan2.
// Generic over the universal-intrinsic float vector type.
template<typename V>
static inline V v_atan2(const V& y, const V& x)
{
CV_INSTRUMENT_REGION();
CALL_HAL(not8u, cv_hal_not8u, src1, step1, dst, step, width, height)
CV_CPU_DISPATCH(not8u, (src1, step1, src2, step2, dst, step, width, height), CV_CPU_DISPATCH_MODES_ALL);
using LT = typename VTraits<V>::lane_type;
const V eps = v_setall_<V>((LT)DBL_EPSILON);
const V z = v_setzero_<V>();
const V p7 = v_setall_<V>((LT)-0.04432655554792128);
const V p5 = v_setall_<V>((LT)0.1555786518463281);
const V p3 = v_setall_<V>((LT)-0.3258083974640975);
const V p1 = v_setall_<V>((LT)0.9997878412794807);
const V vpi2 = v_setall_<V>((LT)(CV_PI/2));
const V vpi = v_setall_<V>((LT)CV_PI);
V ax = v_abs(x), ay = v_abs(y);
V c = v_div(v_min(ax, ay), v_add(v_max(ax, ay), eps));
V c2 = v_mul(c, c);
V a = v_mul(v_fma(v_fma(v_fma(p7, c2, p5), c2, p3), c2, p1), c);
a = v_select(v_ge(ax, ay), a, v_sub(vpi2, a));
a = v_select(v_lt(x, z), v_sub(vpi, a), a);
a = v_select(v_lt(y, z), v_sub(z, a), a);
return a;
}
#undef DEFINE_CMP_OP_DISPATCHER
#define DEFINE_CMP_OP_DISPATCHER(opname, decl_type, type) \
void opname(const decl_type* src1, size_t step1, const decl_type* src2, size_t step2, \
uchar* dst, size_t step, int width, int height, void* params) \
{ \
CV_INSTRUMENT_REGION(); \
CV_CPU_DISPATCH(opname, ((const type*)src1, step1, (const type*)src2, step2, \
dst, step, width, height, *(int*)params), CV_CPU_DISPATCH_MODES_ALL); \
struct EwAtan2 {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_atan2(a, b); } // a = y, b = x
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return std::atan2(a, b); }
};
// hypot(x, y) = sqrt(x^2 + y^2): NAIVE (matches cv::magnitude; overflow at |x| ~ 1e19+ for f32
// inputs is accepted), computed in the float work type; T x T -> T over the float depths.
struct EwHypot {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_sqrt(v_fma(a, a, v_mul(b, b))); }
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return std::sqrt(a*a + b*b); }
};
// min / max / absdiff: T x T -> T (same depth in and out, no scale). v_min/v_max exist for every
// vector lane type (64-bit ints fall back to scalar). absdiff uses v_absdiff (defined for the
// UNSIGNED and float lane types - signed/wide depths go through the scalar path), and the scalar
// |a-b| is computed branch-wise so it never underflows an unsigned work type.
struct EwMin {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_min(a, b); }
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return std::min(a, b); }
};
struct EwMax {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_max(a, b); }
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return std::max(a, b); }
};
struct EwAbsdiff {
static constexpr bool useScalar = false;
// The absdiff RESULT is the UNSIGNED type of the input width: v_absdiff(v_int8/16/32) already returns
// v_uint8/16/32 (the true |a-b|, which can exceed the signed max), and v_absdiff on unsigned/float
// returns the same type. vec() therefore returns THAT type (deduced), not the input V, so the kernel
// stores it through the matching UNSIGNED v_store_pair_as (an honest same-type store) - no reinterpret,
// no touching the saturating-narrow overloads' semantics. (This is why the kernel keeps the vec
// result in its own variable rather than reusing the input operand.)
template<typename V> static auto vec(const V& a, const V& b) { return v_absdiff(a, b); }
template<typename V, typename S> static auto vec(const V& a, const V& b, const S&) { return v_absdiff(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return a > b ? W(a - b) : W(b - a); }
};
// Signed T -> SAME signed T (rdepth==T): |a-b| saturated into the signed range, in ONE pass. cv::absdiff
// keeps the signed depth, so the generic path computes absdiff->unsigned then casts back down (2 insns);
// this fuses it. v_absdiff yields the unsigned |a-b|; v_min clamps to the signed max (already >=0), then
// a same-width reinterpret to signed (all values now fit).
struct EwAbsdiffS {
static constexpr bool useScalar = false;
static v_int8 vec(const v_int8& a, const v_int8& b) { return v_reinterpret_as_s8 (v_min(v_absdiff(a, b), vx_setall_u8 (0x7f))); }
static v_int16 vec(const v_int16& a, const v_int16& b) { return v_reinterpret_as_s16(v_min(v_absdiff(a, b), vx_setall_u16(0x7fff))); }
static v_int32 vec(const v_int32& a, const v_int32& b) { return v_reinterpret_as_s32(v_min(v_absdiff(a, b), vx_setall_u32(0x7fffffff))); }
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return a > b ? W(a - b) : W(b - a); }
};
// compare: T x T -> u8 mask. cmp(a,b) is the scalar relation over a work type W (integers compare
// directly, f16/bf16 through float); vec(a,b) is the vector relation returning an all-ones/zero mask
// in the operand's own lane type. The kernel turns either into 0 / trueVal (1 or 255 via TKernel::flags).
struct EwCmpEq { template<typename W> static bool cmp(W a, W b) { return a == b; }
template<typename V> static V vec(const V& a, const V& b) { return v_eq(a, b); } };
struct EwCmpNe { template<typename W> static bool cmp(W a, W b) { return a != b; }
template<typename V> static V vec(const V& a, const V& b) { return v_ne(a, b); } };
// LT/LE are synthesized from GT/GE via the EW_KERNEL_SWAP01 flag (a<b == b>a), so no Lt/Le kernels.
struct EwCmpGt { template<typename W> static bool cmp(W a, W b) { return a > b; }
template<typename V> static V vec(const V& a, const V& b) { return v_gt(a, b); } };
struct EwCmpGe { template<typename W> static bool cmp(W a, W b) { return a >= b; }
template<typename V> static V vec(const V& a, const V& b) { return v_ge(a, b); } };
// bitwise AND/OR/XOR: bit-pattern op, type-agnostic. Run on the UNSIGNED integer whose width matches
// the element (u8/u16/u32/u64), so one functor set covers every depth. No scale, no widening (T x T ->
// T, exactly like min/max); preproc is the identity (min/max share this shape). 64-bit uses the scalar
// path (no widening vector helpers), the rest ride vecBinaryKernel's native same-type path.
struct EwAnd {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_and(a, b); }
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return W(a & b); }
};
struct EwOr {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_or(a, b); }
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return W(a | b); }
};
struct EwXor {
static constexpr bool useScalar = false;
template<typename V> static V vec(const V& a, const V& b) { return v_xor(a, b); }
template<typename V, typename S> static V vec(const V& a, const V& b, const S&) { return vec(a, b); }
template<typename V> static V preproc(const V& a, const V&) { return a; }
template<typename W, typename ST> static W scl(W a, W b, ST) { return W(a ^ b); }
};
// Collapse a gap-free 2D tile to 1D (call with the per-operand x/y-steps).
#define EW_TRY_COLLAPSE(NSRC) \
if (height > 1 && dsty == (size_t)width && \
s0y == s0x*(size_t)width && (NSRC < 2 || s1y == s1x*(size_t)width)) \
{ width *= height; height = 1; }
// Unified binary kernel: T0 x T1 -> Tr (operands same depth for arithmetic; cast is separate).
// Wvec = work vector. Native (v_uint8/...) drives the same-type saturating path
// (v_add saturates 8/16-bit, wraps 32-bit); v_float32 drives the widening hub.
// WT = scalar work type for the tail.
// Op = operation functor (vec()/scl()).
// use_simd = compile-time switch; false => pure scalar (32/64-bit widened outputs, f64).
// stepx in {0,1}; dst contiguous. In-place safe (see file header).
template<typename T, typename Tr, typename WT, class Op, typename ST=WT>
static int scalarBinaryKernel(const void* src0_, size_t s0y, size_t s0x,
const void* src1_, size_t s1y, size_t s1x,
const void*, size_t, size_t, void* dst_, size_t dsty,
int width, int height, const double* params, int, void*)
{
s0y /= sizeof(T);
s1y /= sizeof(T);
dsty /= sizeof(Tr);
CV_Assert((s0x|s1x) == 1u || (s0x|s1x) + (size_t)width == 1u);
const T* src0 = (const T*)src0_;
const T* src1 = (const T*)src1_;
Tr* dst = (Tr*)dst_;
[[maybe_unused]] ST scalar = saturate_cast<ST>(params[0]); // mul/div scale; ignored by add/sub
EW_TRY_COLLAPSE(2);
for (int y = 0; y < height; y++, src0 += s0y, src1 += s1y, dst += dsty)
{
if (s0x == s1x) {
for (int x = 0; x < width; x++)
dst[x] = saturate_cast<Tr>(Op::scl((WT)src0[x], (WT)src1[x], scalar));
}
else if (s0x == 0) {
WT sc0 = (WT)src0[0];
for (int x = 0; x < width; x++)
dst[x] = saturate_cast<Tr>(Op::scl(sc0, (WT)src1[x], scalar));
}
else {
WT sc1 = (WT)src1[0];
for (int x = 0; x < width; x++)
dst[x] = saturate_cast<Tr>(Op::scl((WT)src0[x], sc1, scalar));
}
}
return 0;
}
DEFINE_CMP_OP_DISPATCHER(cmp8u, uchar, uchar)
DEFINE_CMP_OP_DISPATCHER(cmp8s, schar, schar)
DEFINE_CMP_OP_DISPATCHER(cmp16u, ushort, ushort)
DEFINE_CMP_OP_DISPATCHER(cmp16s, short, short)
DEFINE_CMP_OP_DISPATCHER(cmp32u, unsigned, unsigned)
DEFINE_CMP_OP_DISPATCHER(cmp32s, int, int)
DEFINE_CMP_OP_DISPATCHER(cmp64u, uint64, uint64)
DEFINE_CMP_OP_DISPATCHER(cmp64s, int64, int64)
DEFINE_CMP_OP_DISPATCHER(cmp16f, cv_hal_f16, hfloat)
DEFINE_CMP_OP_DISPATCHER(cmp16bf, cv_hal_bf16, bfloat)
DEFINE_CMP_OP_DISPATCHER(cmp32f, float, float)
DEFINE_CMP_OP_DISPATCHER(cmp64f, double, double)
#undef DEFINE_BINARY_OP_W_PARAMS_DISPATCHER
#define DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname, decl_type, type, read_params, paramname) \
void opname(const decl_type* src1, size_t step1, const decl_type* src2, size_t step2, \
decl_type* dst, size_t step, int width, int height, void* params_) \
{ \
CV_INSTRUMENT_REGION(); \
read_params; \
CALL_HAL(opname, cv_hal_##opname, src1, step1, src2, step2, dst, step, width, height, paramname) \
CV_CPU_DISPATCH(opname, ((const type*)src1, step1, (const type*)src2, step2, \
(type*)dst, step, width, height, paramname), CV_CPU_DISPATCH_MODES_ALL); \
template<typename T, typename WT>
static void expandScalar(const T* sc, size_t sx, int n0, WT* scbuf, int n)
{
int i = 0;
for (; i < n0; i++) scbuf[i] = (WT)sc[i*sx];
for (; i < n; i++) scbuf[i] = scbuf[i - n0];
}
#undef DEFINE_BINARY_OP_W_PARAMS_DISPATCHER_ALLTYPES
#define DEFINE_BINARY_OP_W_PARAMS_DISPATCHER_ALLTYPES(opname, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##8u, uchar, uchar, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##8s, schar, schar, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##16u, ushort, ushort, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##16s, short, short, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##32u, unsigned, unsigned, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##32s, int, int, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##64u, uint64, uint64, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##64s, int64, int64, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##16f, cv_hal_f16, hfloat, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##16bf, cv_hal_bf16, bfloat, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##32f, float, float, read_params, paramname) \
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER(opname##64f, double, double, read_params, paramname)
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER_ALLTYPES(mul, double scale = *(double*)params_, scale)
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER_ALLTYPES(div, double scale = *(double*)params_, scale)
DEFINE_BINARY_OP_W_PARAMS_DISPATCHER_ALLTYPES(addWeighted, \
double w[3]; \
w[0]=((double*)params_)[0]; \
w[1]=((double*)params_)[1]; \
w[2]=((double*)params_)[2];, \
w)
#undef DEFINE_UNARY_OP_W_PARAMS_DISPATCHER
#define DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(opname, decl_type, type, read_params, paramname) \
void opname(const decl_type* src1, size_t step1, const decl_type*, size_t, \
decl_type* dst, size_t step, int width, int height, void* params_) \
{ \
CV_INSTRUMENT_REGION(); \
read_params; \
CALL_HAL(opname, cv_hal_##opname, src1, step1, dst, step, width, height, paramname) \
CV_CPU_DISPATCH(opname, ((const type*)src1, step1, nullptr, 0, \
(type*)dst, step, width, height, paramname), CV_CPU_DISPATCH_MODES_ALL); \
// Decode the per-channel patch bytes (mask + value) from the kernel flags into small arrays. Uniform
// (no EW_CMP_PATCH): mask = trueVal, value = 0 (an ordinary compare). Per-channel: from the 4-bit
// fields. The compare kernels apply result = (rawmask & mask) | value per channel - folding the
// former separate patch pass into the compare (one pass).
static inline void cmpUnpackPatch(int flags, uchar trueVal, uchar mvals[4], uchar vvals[4])
{
if (flags & EW_CMP_PATCH)
for (int c = 0; c < 4; c++)
{
const int f = (flags >> (EW_CMP_PATCH_SHIFT + c*4)) & 0xF;
mvals[c] = (uchar)cmpPatchByte(f & 3);
vvals[c] = (uchar)cmpPatchByte((f >> 2) & 3);
}
else
for (int c = 0; c < 4; c++) { mvals[c] = trueVal; vvals[c] = 0; }
}
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip8u, uchar, uchar, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip8s, schar, schar, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip16u, ushort, ushort, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip16s, short, short, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip32u, unsigned, unsigned, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip32s, int, int, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip64u, uint64, uint64, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip64s, int64, int64, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip16f, cv_hal_f16, hfloat, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip16bf, cv_hal_bf16, bfloat, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip32f, float, float, double scale = *(double*)params_, scale)
DEFINE_UNARY_OP_W_PARAMS_DISPATCHER(recip64f, double, double, double scale = *(double*)params_, scale)
// Scalar comparison kernel: T x T -> u8 mask. Result is 0 (false) or `trueVal` (true), trueVal read
// from TKernel::flags - 255 (default, matching cv::compare) or 1 (a numpy-style 0/1 mask). Integers
// compare directly (WT == T); f16/bf16 compare through a float WT. This is the fallback for the depths
// the SIMD kernel doesn't cover (f16/bf16/64-bit). stepx in {0,1}; dst is u8, byte step == element step.
template<typename T, typename WT, class Cmp>
static int scalarCompareKernel(const void* src0_, size_t s0y, size_t s0x,
const void* src1_, size_t s1y, size_t s1x,
const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int flags, void*)
{
// LT/LE reuse the GT/GE kernel with the operands swapped (a<b == b>a, a<=b == b>=a).
if (flags & EW_KERNEL_SWAP01) { std::swap(src0_, src1_); std::swap(s0y, s1y); std::swap(s0x, s1x); }
s0y /= sizeof(T);
s1y /= sizeof(T);
CV_Assert((s0x|s1x) == 1u || (s0x|s1x) + (size_t)width == 1u);
const T* src0 = (const T*)src0_;
const T* src1 = (const T*)src1_;
uchar* dst = (uchar*)dst_;
const uchar trueVal = (flags & EW_KERNEL_MASK1) ? 1 : 255;
uchar mvals[4], vvals[4];
cmpUnpackPatch(flags, trueVal, mvals, vvals);
const bool perch = (flags & EW_CMP_PATCH) != 0; // per-channel fix-up (width=cn<=4)
CV_Assert(!perch || width <= 4); // a per-channel patch is a short-row tile
#define EW_CMP_APPLY(cond, x) (perch ? (uchar)(((cond) ? mvals[x] : 0) | vvals[x]) \
: (uchar)((cond) ? trueVal : 0))
EW_TRY_COLLAPSE(2);
for (int y = 0; y < height; y++, src0 += s0y, src1 += s1y, dst += dsty)
{
if (s0x == s1x) {
for (int x = 0; x < width; x++) dst[x] = EW_CMP_APPLY(Cmp::cmp((WT)src0[x], (WT)src1[x]), x);
}
else if (s0x == 0) {
WT a = (WT)src0[0];
for (int x = 0; x < width; x++) dst[x] = EW_CMP_APPLY(Cmp::cmp(a, (WT)src1[x]), x);
}
else {
WT b = (WT)src1[0];
for (int x = 0; x < width; x++) dst[x] = EW_CMP_APPLY(Cmp::cmp((WT)src0[x], b), x);
}
}
#undef EW_CMP_APPLY
return 0;
}
// SIMD comparison kernel: T x T -> u8 mask, for the directly-comparable depths (u8/s8/u16/s16/u32/
// s32/f32). vec(a,b) gives an all-ones/zero mask in the operand's lane type; v_reinterpret_as turns it
// into the unsigned int of the same width; cmpFuse packs to u8 and applies (rawmask & M) | V. The
// per-row SIMD body runs when an operand is contiguous or broadcast; a per-channel patch (M/V differ
// by channel) only ever arrives as a short-row (width=cn) tile and is handled there.
template<typename T, typename Vvec, typename Uvec, class Cmp>
static int vecCompareKernel(const void* src0_, size_t s0y, size_t s0x,
const void* src1_, size_t s1y, size_t s1x,
const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int flags, void*)
{
// LT/LE reuse the GT/GE kernel with the operands swapped (a<b == b>a, a<=b == b>=a).
if (flags & EW_KERNEL_SWAP01) { std::swap(src0_, src1_); std::swap(s0y, s1y); std::swap(s0x, s1x); }
s0y /= sizeof(T);
s1y /= sizeof(T);
CV_Assert((s0x|s1x) == 1u || (s0x|s1x) + (size_t)width == 1u);
const T* src0 = (const T*)src0_;
const T* src1 = (const T*)src1_;
uchar* dst = (uchar*)dst_;
const uchar trueVal = (flags & EW_KERNEL_MASK1) ? 1 : 255;
uchar mvals[4], vvals[4]; // per-channel fix-up: (rawmask & m) | v
cmpUnpackPatch(flags, trueVal, mvals, vvals);
const bool perch = (flags & EW_CMP_PATCH) != 0;
CV_Assert(!perch || width <= 4); // a per-channel patch is a short-row tile
EW_TRY_COLLAPSE(2);
int y = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
// Short rows (2/3/4 elements): a per-channel scalar over a multi-channel image arrives as a
// (width=cn) x (height=pixels) tile with the scalar broadcast over rows (s?y==0). The per-row SIMD
// below never triggers at such a tiny width, so expand the width<=4 broadcast operand (threshold
// AND the interleaved per-channel mask/value bytes) across the lanes and compare many rows at once.
// This is also the ONLY path a per-channel patch (M/V differ by channel) reaches.
// constexpr: the short-row loads T natively (vx_load->Vvec), so it exists ONLY at native width. The
// widened f16/bf16->f32 path (sizeof(T) < Vvec lane) and 8-byte f64 fall through to per-row + scalar.
// (sizeof(T) <= 4 is part of the constexpr gate: for f64 the body would compile to nothing but
// dead stores - gcc 9 flags scbuf/bsrc/bsx as set-but-not-used there.)
if constexpr (sizeof(T) == sizeof(typename VTraits<Vvec>::lane_type) && sizeof(T) <= 4)
if (height > 1 && width <= 4 &&
((s0y == 0 && s1y == (size_t)width*s1x) || (s1y == 0 && s0y == (size_t)width*s0x)) &&
dsty == (size_t)width)
{
const int VECSZ = VTraits<Vvec>::vlanes(); // sizeof(T)-type lanes
const int VECSZ8 = VTraits<v_uint8>::vlanes(); // u8 output lanes
constexpr int MAXV8 = VTraits<v_uint8>::max_nlanes;
T scbuf[MAXV8 * 3] = {}; // interleaved threshold (elements)
uchar mbuf[MAXV8 * 3] = {}, vbuf[MAXV8 * 3] = {}; // interleaved mask / value (bytes)
// ^ the {} inits are for -Wmaybe-uninitialized only: expandScalar/decode fill every lane that
// is later read, but the compiler cannot prove it with a runtime VECSZ
const bool bc0 = (s0y == 0); // src0 is the broadcast (short) operand
const T* bsrc = bc0 ? src0 : src1; const size_t bsx = bc0 ? s0x : s1x;
const T*& src = bc0 ? src1 : src0; // the row-stepping operand (advanced below)
const size_t sy = bc0 ? s1y : s0y;
// The compare direction (scalar-first when bc0, array-first otherwise) is hoisted OUT of the row
// loop with an explicit if(bc0) - the loop is written twice so no per-iteration branch remains.
// Per branch: own unroll, threshold + interleaved M/V loaded ONCE. sizeof==1 keeps 12 vectors.
#define EW_CMPR(r, a, b) v_reinterpret_as(Cmp::vec(a, b), r) // rawmask -> sizeof(T)-byte uint
if constexpr (sizeof(T) == 1) // 3 u8 masks -> 3 u8 stores
{
const int ewidth = VECSZ8 * 3;
expandScalar(bsrc, bsx, width, scbuf, ewidth);
expandScalar(mvals, (size_t)1, width, mbuf, ewidth);
expandScalar(vvals, (size_t)1, width, vbuf, ewidth);
const int dy = ewidth / width;
Vvec sc0=vx_load(scbuf), sc1=vx_load(scbuf+VECSZ), sc2=vx_load(scbuf+VECSZ*2);
v_uint8 M0=vx_load(mbuf), M1=vx_load(mbuf+VECSZ8), M2=vx_load(mbuf+VECSZ8*2);
v_uint8 V0=vx_load(vbuf), V1=vx_load(vbuf+VECSZ8), V2=vx_load(vbuf+VECSZ8*2);
#define EW_ROW1(A0,B0,A1,B1,A2,B2) \
for (; y + dy <= height; y += dy, src += sy*dy, dst += dsty*dy) { \
Uvec r0,r1,r2; EW_CMPR(r0,A0,B0); EW_CMPR(r1,A1,B1); EW_CMPR(r2,A2,B2); \
v_uint8 o0=v_or(v_and(r0,M0),V0),o1=v_or(v_and(r1,M1),V1),o2=v_or(v_and(r2,M2),V2); \
v_store(dst, o0); v_store(dst+VECSZ8, o1); v_store(dst+VECSZ8*2, o2); }
if (bc0) EW_ROW1(sc0,vx_load(src), sc1,vx_load(src+VECSZ), sc2,vx_load(src+VECSZ*2))
else EW_ROW1(vx_load(src),sc0, vx_load(src+VECSZ),sc1, vx_load(src+VECSZ*2),sc2)
#undef EW_ROW1
}
else if constexpr (sizeof(T) == 2) // 3 u16 masks -> 1 full + 1 half u8 store
{
const int ewidth = VECSZ * 3;
expandScalar(bsrc, bsx, width, scbuf, ewidth);
expandScalar(mvals, (size_t)1, width, mbuf, ewidth);
expandScalar(vvals, (size_t)1, width, vbuf, ewidth);
const int dy = ewidth / width;
Vvec sc0=vx_load(scbuf), sc1=vx_load(scbuf+VECSZ), sc2=vx_load(scbuf+VECSZ*2);
v_uint8 M0=vx_load(mbuf), M1=vx_load(mbuf+VECSZ8);
v_uint8 V0=vx_load(vbuf), V1=vx_load(vbuf+VECSZ8);
#define EW_ROW2(A0,B0,A1,B1,A2,B2) \
for (; y + dy <= height; y += dy, src += sy*dy, dst += dsty*dy) { \
Uvec r0,r1,r2; EW_CMPR(r0,A0,B0); EW_CMPR(r1,A1,B1); EW_CMPR(r2,A2,B2); \
v_uint8 b0=v_pack(r0,r1), b1=v_pack(r2,r2); \
v_uint8 o0=v_or(v_and(b0,M0),V0), o1=v_or(v_and(b1,M1),V1); \
v_store(dst, o0); v_store_low(dst+VECSZ8, o1); }
if (bc0) EW_ROW2(sc0,vx_load(src), sc1,vx_load(src+VECSZ), sc2,vx_load(src+VECSZ*2))
else EW_ROW2(vx_load(src),sc0, vx_load(src+VECSZ),sc1, vx_load(src+VECSZ*2),sc2)
#undef EW_ROW2
}
else if constexpr (sizeof(T) == 4) // sizeof==4: 6 u32 -> 3 u16 -> 1 full + 1 half u8
{
const int ewidth = VECSZ * 6;
expandScalar(bsrc, bsx, width, scbuf, ewidth);
expandScalar(mvals, (size_t)1, width, mbuf, ewidth);
expandScalar(vvals, (size_t)1, width, vbuf, ewidth);
const int dy = ewidth / width;
Vvec sc0=vx_load(scbuf),sc1=vx_load(scbuf+VECSZ),sc2=vx_load(scbuf+VECSZ*2),
sc3=vx_load(scbuf+VECSZ*3),sc4=vx_load(scbuf+VECSZ*4),sc5=vx_load(scbuf+VECSZ*5);
v_uint8 M0=vx_load(mbuf), M1=vx_load(mbuf+VECSZ8);
v_uint8 V0=vx_load(vbuf), V1=vx_load(vbuf+VECSZ8);
#define EW_ROW4(A0,B0,A1,B1,A2,B2,A3,B3,A4,B4,A5,B5) \
for (; y + dy <= height; y += dy, src += sy*dy, dst += dsty*dy) { \
Uvec r0,r1,r2,r3,r4,r5; \
EW_CMPR(r0,A0,B0); EW_CMPR(r1,A1,B1); EW_CMPR(r2,A2,B2); \
EW_CMPR(r3,A3,B3); EW_CMPR(r4,A4,B4); EW_CMPR(r5,A5,B5); \
v_uint16 p0=v_pack(r0,r1), p1=v_pack(r2,r3), p2=v_pack(r4,r5); \
v_uint8 q0=v_pack(p0,p1), q1=v_pack(p2,p2); \
v_uint8 o0=v_or(v_and(q0,M0),V0), o1=v_or(v_and(q1,M1),V1); \
v_store(dst, o0); v_store_low(dst+VECSZ8, o1); }
if (bc0) EW_ROW4(sc0,vx_load(src),sc1,vx_load(src+VECSZ),sc2,vx_load(src+VECSZ*2),
sc3,vx_load(src+VECSZ*3),sc4,vx_load(src+VECSZ*4),sc5,vx_load(src+VECSZ*5))
else EW_ROW4(vx_load(src),sc0,vx_load(src+VECSZ),sc1,vx_load(src+VECSZ*2),sc2,
vx_load(src+VECSZ*3),sc3,vx_load(src+VECSZ*4),sc4,vx_load(src+VECSZ*5),sc5)
#undef EW_ROW4
}
#undef EW_CMPR
}
#endif
for (; y < height; y++, src0 += s0y, src1 += s1y, dst += dsty)
{
int x = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
// SIMD for BOTH the contiguous case AND a broadcast operand (a per-channel scalar const has
// step 0 - without this the scalar-vs-array compare, incl. every multi-channel scalar compare,
// fell to the scalar tail below, ~5x slower). vx_setall broadcasts the step-0 operand once.
// 4-vector unroll + halide right-edge backoff (reprocess the last 4*VECSZ when width is not a
// multiple); dst is a separate mask buffer so the overlap is a harmless idempotent rewrite.
const int VECSZ = VTraits<Vvec>::vlanes();
const int VECSZ8 = VTraits<v_uint8>::vlanes();
v_uint8 vT; v_setall_mask(vT, trueVal); // uniform mask (per-channel patch is short-row)
const bool tailTrick = width >= 4*VECSZ && src0_ != dst_ && src1_ != dst_;
// Narrow the 4 masks to u8 FIRST, then apply trueVal on the u8 result (1 and per u8 vector, not
// per wide lane-group). One block per MASK lane width (== sizeof(T) natively, but 4 for the
// f16/bf16->f32 widened path); a 4-vector unroll spans exactly 4*VECSZ elements.
constexpr size_t MW = sizeof(typename VTraits<Uvec>::lane_type);
#define PACK_STORE_CMP_RESULT(m0,m1,m2,m3, D) do { \
if constexpr (MW == 1u) { \
v_uint8 o0=v_and(m0,vT),o1=v_and(m1,vT),o2=v_and(m2,vT),o3=v_and(m3,vT); \
v_store((D),o0); v_store((D)+VECSZ8,o1); v_store((D)+VECSZ8*2,o2); v_store((D)+VECSZ8*3,o3); \
} else if constexpr (MW == 2u) { \
v_uint8 o0=v_and(v_pack(m0,m1),vT), o1=v_and(v_pack(m2,m3),vT); \
v_store((D),o0); v_store((D)+VECSZ8,o1); \
} else if constexpr (MW == 4u) { \
v_uint8 o0=v_and(v_pack(v_pack(m0,m1),v_pack(m2,m3)),vT); \
v_store((D),o0); \
} else { /* sizeof==8: 4 u64 -> 2 u32 -> 1 u16 -> u8 (low half), 4*VECSZ elems -> store_low */ \
v_uint16 g=v_pack(v_pack(m0,m1),v_pack(m2,m3)); \
v_uint8 o0=v_and(v_pack(g,g),vT); \
v_store_low((D),o0); \
} } while(0)
if (s0x == 1u && s1x == 1u)
{
for (; x < width; x += 4*VECSZ)
{
if (x + 4*VECSZ > width) { if (!tailTrick) break; x = width - 4*VECSZ; }
Vvec a0, a1, a2, a3, b0, b1, b2, b3; // vx_load_pair_as widens f16/bf16->f32, else native
vx_load_pair_as(src0+x, a0, a1); vx_load_pair_as(src0+x+2*VECSZ, a2, a3);
vx_load_pair_as(src1+x, b0, b1); vx_load_pair_as(src1+x+2*VECSZ, b2, b3);
Uvec m0, m1, m2, m3;
v_reinterpret_as(Cmp::vec(a0, b0), m0); v_reinterpret_as(Cmp::vec(a1, b1), m1);
v_reinterpret_as(Cmp::vec(a2, b2), m2); v_reinterpret_as(Cmp::vec(a3, b3), m3);
PACK_STORE_CMP_RESULT(m0, m1, m2, m3, dst+x);
}
}
else if (s1x == 0u) // src1 (e.g. the scalar) broadcast
{
Vvec b0; vx_setall_as(src1, b0);
for (; x < width; x += 4*VECSZ)
{
if (x + 4*VECSZ > width) { if (!tailTrick) break; x = width - 4*VECSZ; }
Vvec a0, a1, a2, a3;
vx_load_pair_as(src0+x, a0, a1); vx_load_pair_as(src0+x+2*VECSZ, a2, a3);
Uvec m0, m1, m2, m3;
v_reinterpret_as(Cmp::vec(a0, b0), m0); v_reinterpret_as(Cmp::vec(a1, b0), m1);
v_reinterpret_as(Cmp::vec(a2, b0), m2); v_reinterpret_as(Cmp::vec(a3, b0), m3);
PACK_STORE_CMP_RESULT(m0, m1, m2, m3, dst+x);
}
}
else if (s0x == 0u) // src0 broadcast
{
Vvec a0; vx_setall_as(src0, a0);
for (; x < width; x += 4*VECSZ)
{
if (x + 4*VECSZ > width) { if (!tailTrick) break; x = width - 4*VECSZ; }
Vvec b0, b1, b2, b3;
vx_load_pair_as(src1+x, b0, b1); vx_load_pair_as(src1+x+2*VECSZ, b2, b3);
Uvec m0, m1, m2, m3;
v_reinterpret_as(Cmp::vec(a0, b0), m0); v_reinterpret_as(Cmp::vec(a0, b1), m1);
v_reinterpret_as(Cmp::vec(a0, b2), m2); v_reinterpret_as(Cmp::vec(a0, b3), m3);
PACK_STORE_CMP_RESULT(m0, m1, m2, m3, dst+x);
}
}
#undef PACK_STORE_CMP_RESULT
#endif
// Fold the fix-up: raw all-ones/zero mask -> (raw & M) | V. Uniform => raw & trueVal (V=0); a
// per-channel patch (width=cn<=4 here) indexes M/V by the channel x.
#define EW_CMP_APPLY(cond, x) (perch ? (uchar)(((cond) ? mvals[x] : 0) | vvals[x]) \
: (uchar)((cond) ? trueVal : 0))
if (s0x == s1x) {
for (; x < width; x++) dst[x] = EW_CMP_APPLY(Cmp::cmp(src0[x], src1[x]), x);
}
else if (s0x == 0) {
T a = src0[0];
for (; x < width; x++) dst[x] = EW_CMP_APPLY(Cmp::cmp(a, src1[x]), x);
}
else {
T b = src1[0];
for (; x < width; x++) dst[x] = EW_CMP_APPLY(Cmp::cmp(src0[x], b), x);
}
#undef EW_CMP_APPLY
}
return 0;
}
// Unified binary kernel: T0 x T1 -> Tr (operands same depth for arithmetic; cast is separate).
// Wvec = work vector. Native (v_uint8/...) drives the same-type saturating path
// (v_add saturates 8/16-bit, wraps 32-bit); v_float32 drives the widening hub.
// WT = scalar work type for the tail.
// Op = operation functor (vec()/scl()).
// use_simd = compile-time switch; false => pure scalar (32/64-bit widened outputs, f64).
// stepx in {0,1}; dst contiguous. In-place safe (see file header).
template<typename T, typename Tr, typename Wvec, typename WT, class Op, typename ST=WT, typename Wvec1=Wvec>
static int vecBinaryKernel(const void* src0_, size_t s0y, size_t s0x,
const void* src1_, size_t s1y, size_t s1x,
const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double* params, int, void*)
{
s0y /= sizeof(T);
s1y /= sizeof(T);
dsty /= sizeof(Tr);
CV_Assert((s0x|s1x) == 1u || (s0x|s1x) + (size_t)width == 1u);
const T* src0 = (const T*)src0_;
const T* src1 = (const T*)src1_;
Tr* dst = (Tr*)dst_;
[[maybe_unused]] ST scalar = saturate_cast<ST>(params[0]); // mul/div scale; ignored by add/sub
EW_TRY_COLLAPSE(2);
int y = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
using Wlane = typename VTraits<Wvec>::lane_type;
const int VECSZ = VTraits<Wvec>::vlanes();
const bool use_tail_trick = width >= VECSZ*4 && src0_ != dst_ && src1_ != dst_;
[[maybe_unused]] Wvec vscalar;
vx_setall_as(&scalar, vscalar);
if (height > 1 && width <= 4 &&
((s0y == 0 && s1y == width*s1x) ||
(s1y == 0 && s0y == width*s0x)) &&
dsty == (size_t)width) {
constexpr int MAXVECSZ = VTraits<Wvec>::max_nlanes;
Wlane scbuf[MAXVECSZ*6] = {}; // {} for -Wmaybe-uninitialized only (filled up to VECSZ*6)
const int ewidth = VECSZ*6;
expandScalar(s0y == 0 ? src0 : src1, s0y == 0 ? s0x : s1x, width, scbuf, ewidth);
int dy = ewidth / width;
Wvec sc0, sc1, sc2, sc3, sc4, sc5;
sc0 = Op::preproc(vx_load(scbuf), vscalar);
sc1 = Op::preproc(vx_load(scbuf + VECSZ), vscalar);
sc2 = Op::preproc(vx_load(scbuf + VECSZ*2), vscalar);
sc3 = Op::preproc(vx_load(scbuf + VECSZ*3), vscalar);
sc4 = Op::preproc(vx_load(scbuf + VECSZ*4), vscalar);
sc5 = Op::preproc(vx_load(scbuf + VECSZ*5), vscalar);
if (s0y == 0) {
for (; y + dy <= height; y += dy, src1 += s1y*dy, dst += dsty*dy) {
Wvec v0, v1, v2, v3, v4, v5;
vx_load_pair_as(src1, v0, v1);
vx_load_pair_as(src1 + VECSZ*2, v2, v3);
vx_load_pair_as(src1 + VECSZ*4, v4, v5);
auto w0 = Op::vec(sc0, v0, vscalar), w1 = Op::vec(sc1, v1, vscalar),
w2 = Op::vec(sc2, v2, vscalar), w3 = Op::vec(sc3, v3, vscalar),
w4 = Op::vec(sc4, v4, vscalar), w5 = Op::vec(sc5, v5, vscalar);
v_store_pair_as(dst, w0, w1);
v_store_pair_as(dst + VECSZ*2, w2, w3);
v_store_pair_as(dst + VECSZ*4, w4, w5);
}
}
else {
for (; y + dy <= height; y += dy, src0 += s0y*dy, dst += dsty*dy) {
Wvec v0, v1, v2, v3, v4, v5;
vx_load_pair_as(src0, v0, v1);
vx_load_pair_as(src0 + VECSZ*2, v2, v3);
vx_load_pair_as(src0 + VECSZ*4, v4, v5);
auto w0 = Op::vec(v0, sc0, vscalar), w1 = Op::vec(v1, sc1, vscalar),
w2 = Op::vec(v2, sc2, vscalar), w3 = Op::vec(v3, sc3, vscalar),
w4 = Op::vec(v4, sc4, vscalar), w5 = Op::vec(v5, sc5, vscalar);
v_store_pair_as(dst, w0, w1);
v_store_pair_as(dst + VECSZ*2, w2, w3);
v_store_pair_as(dst + VECSZ*4, w4, w5);
}
}
}
#endif
for (; y < height; y++, src0 += s0y, src1 += s1y, dst += dsty)
{
int x = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
Wvec a0, a1, a2, a3, b0, b1, b2, b3;
if (s0x == s1x) {
if (!Op::useScalar || scalar == ST(1)) {
// this branch runs on Wvec1, which may be WIDER-laned than Wvec (e.g. the u8 mul
// path: Wvec1=v_uint16 is 2x the lanes of Wvec=v_float32) - step by ITS lane count,
// or the pairs overlap and the tail backoff writes past the row end
const int VECSZ1 = VTraits<Wvec1>::vlanes();
const bool tail_trick1 = width >= VECSZ1*4 && src0_ != dst_ && src1_ != dst_;
for (; x < width; x += VECSZ1*4) {
Wvec1 a0_, a1_, a2_, a3_, b0_, b1_, b2_, b3_;
if (x + VECSZ1*4 > width) { if (!tail_trick1) break; x = width - VECSZ1*4; }
vx_load_pair_as(src0 + x, a0_, a1_);
vx_load_pair_as(src0 + x + VECSZ1*2, a2_, a3_);
vx_load_pair_as(src1 + x, b0_, b1_);
vx_load_pair_as(src1 + x + VECSZ1*2, b2_, b3_);
auto c0 = Op::vec(a0_, b0_), c1 = Op::vec(a1_, b1_),
c2 = Op::vec(a2_, b2_), c3 = Op::vec(a3_, b3_);
v_store_pair_as(dst + x, c0, c1);
v_store_pair_as(dst + x + VECSZ1*2, c2, c3);
}
}
else {
for (; x < width; x += VECSZ*4) {
if (x + VECSZ*4 > width) { if (!use_tail_trick) break; x = width - VECSZ*4; }
vx_load_pair_as(src0 + x, a0, a1);
vx_load_pair_as(src0 + x + VECSZ*2, a2, a3);
vx_load_pair_as(src1 + x, b0, b1);
vx_load_pair_as(src1 + x + VECSZ*2, b2, b3);
auto c0 = Op::vec(Op::preproc(a0, vscalar), b0, vscalar),
c1 = Op::vec(Op::preproc(a1, vscalar), b1, vscalar),
c2 = Op::vec(Op::preproc(a2, vscalar), b2, vscalar),
c3 = Op::vec(Op::preproc(a3, vscalar), b3, vscalar);
v_store_pair_as(dst + x, c0, c1);
v_store_pair_as(dst + x + VECSZ*2, c2, c3);
}
}
}
else if (s1x == 0) {
vx_setall_as(src1, b0);
b0 = Op::preproc(b0, vscalar);
for (; x < width; x += VECSZ*4) {
if (x + VECSZ*4 > width) { if (!use_tail_trick) break; x = width - VECSZ*4; }
vx_load_pair_as(src0 + x, a0, a1);
vx_load_pair_as(src0 + x + VECSZ*2, a2, a3);
auto c0 = Op::vec(a0, b0, vscalar), c1 = Op::vec(a1, b0, vscalar),
c2 = Op::vec(a2, b0, vscalar), c3 = Op::vec(a3, b0, vscalar);
v_store_pair_as(dst + x, c0, c1);
v_store_pair_as(dst + x + VECSZ*2, c2, c3);
}
}
else {
vx_setall_as(src0, b0);
b0 = Op::preproc(b0, vscalar);
for (; x < width; x += VECSZ*4) {
if (x + VECSZ*4 > width) { if (!use_tail_trick) break; x = width - VECSZ*4; }
vx_load_pair_as(src1 + x, a0, a1);
vx_load_pair_as(src1 + x + VECSZ*2, a2, a3);
auto c0 = Op::vec(b0, a0, vscalar), c1 = Op::vec(b0, a1, vscalar),
c2 = Op::vec(b0, a2, vscalar), c3 = Op::vec(b0, a3, vscalar);
v_store_pair_as(dst + x, c0, c1);
v_store_pair_as(dst + x + VECSZ*2, c2, c3);
}
}
#endif
for (; x < width; x++)
dst[x] = saturate_cast<Tr>(Op::scl((WT)src0[x*s0x], (WT)src1[x*s1x], scalar));
}
#if (CV_SIMD || CV_SIMD_SCALABLE)
vx_cleanup();
#endif
return 0;
}
// ===========================================================================
// OP_ADD
// ===========================================================================
// add dispatch: T x T -> R (operands already same depth T).
// - native saturating path (Wvec = native, use_simd=true) for T -> T on <=32-bit ints + f32
// (32-bit lanes via the local v_add_sat/v_sub_sat - v_add/v_sub wrap there);
// - widening f32-hub (Wvec = v_float32, use_simd=true) for the widened/half/float outputs;
// - scalar (use_simd=false) for 64-bit results and f64 (no vector path yet).
template<class Op>
TKernel getAddSubFunc(int T, int R)
{
KernelFunc fptr = nullptr;
switch (T)
{
case CV_8U:
fptr = R == CV_8U ? vecBinaryKernel<uchar, uchar, v_uint8, short, Op, uchar> :
R == CV_16S ? vecBinaryKernel<uchar, short, v_int16, short, Op, short> :
R == CV_32S ? vecBinaryKernel<uchar, int, v_int16, short, Op, short> :
R == CV_32F ? vecBinaryKernel<uchar, float, v_int16, short, Op, short> : nullptr;
break;
case CV_8S:
fptr =
R == CV_8S ? vecBinaryKernel<schar, schar, v_int8, short, Op, schar> :
R == CV_16S ? vecBinaryKernel<schar, short, v_int16, short, Op, short> :
R == CV_32S ? vecBinaryKernel<schar, int, v_int16, short, Op, short> :
R == CV_32F ? vecBinaryKernel<schar, float, v_int16, short, Op, short> : nullptr;
break;
case CV_16U:
fptr = R == CV_16U ? vecBinaryKernel<ushort, ushort, v_uint16, int, Op, ushort> :
R == CV_32S ? vecBinaryKernel<ushort, int, v_int32, int, Op, int> :
R == CV_32F ? vecBinaryKernel<ushort, float, v_int32, int, Op, int> : nullptr;
break;
case CV_16S:
fptr = R == CV_16S ? vecBinaryKernel<short, short, v_int16, int, Op, short> :
R == CV_32S ? vecBinaryKernel<short, int, v_int32, int, Op, int> :
R == CV_32F ? vecBinaryKernel<short, float, v_int32, int, Op, int> : nullptr;
break;
case CV_32U:
fptr = R == CV_32U ? vecBinaryKernel<unsigned, unsigned, v_uint32, int64_t, Op, unsigned> :
R == CV_64S ? scalarBinaryKernel<unsigned, int64_t, int64_t, Op> :
R == CV_64F ? scalarBinaryKernel<unsigned, double, int64_t, Op> : nullptr;
break;
case CV_32S:
fptr = R == CV_32S ? vecBinaryKernel<int, int, v_int32, int64_t, Op, int> :
R == CV_64S ? scalarBinaryKernel<int, int64_t, int64_t, Op> :
R == CV_64F ? scalarBinaryKernel<int, double, int64_t, Op> : nullptr;
break;
case CV_64U:
fptr = R == CV_64U ? scalarBinaryKernel<uint64_t, uint64_t, uint64_t, Op> :
R == CV_64F ? scalarBinaryKernel<uint64_t, double, double, Op> : nullptr;
break;
case CV_64S:
fptr = R == CV_64S ? scalarBinaryKernel<int64_t, int64_t, int64_t, Op> :
R == CV_64F ? scalarBinaryKernel<int64_t, double, double, Op> : nullptr;
break;
case CV_16F:
fptr =
#if CV_SIMD_16F
R == CV_16F ? vecBinaryKernel<hfloat, hfloat, v_float16, float, Op, hfloat> :
#else
R == CV_16F ? vecBinaryKernel<hfloat, hfloat, v_float32, float, Op, float> :
#endif
R == CV_32F ? vecBinaryKernel<hfloat, float, v_float32, float, Op, float> : nullptr;
break;
case CV_16BF:
fptr =
R == CV_16BF ? vecBinaryKernel<bfloat, bfloat, v_float32, float, Op, float> :
R == CV_32F ? vecBinaryKernel<bfloat, float, v_float32, float, Op, float> : nullptr;
break;
case CV_32F:
fptr = R == CV_32F ? vecBinaryKernel<float, float, v_float32, float, Op, float> : nullptr;
break;
case CV_64F:
#if CV_SIMD_64F
fptr = R == CV_64F ? vecBinaryKernel<double, double, v_float64, double, Op> : nullptr;
#else
fptr = R == CV_64F ? scalarBinaryKernel<double, double, double, Op> : nullptr;
#endif
break;
default:
;
}
return {fptr, nullptr, 0};
}
TKernel getMulFunc_(int T, int R)
{
KernelFunc fptr = nullptr;
switch (T)
{
case CV_8U:
fptr = // scale==1 fast path: whole u8 registers, v_mul_sat widens+clamps inside;
// Wvec (f16 where available, else f32) for the scale path
#if CV_SIMD_16F
R == CV_8U ? vecBinaryKernel<uchar, uchar, v_float16, float, EwMul, float, v_uint8> :
#else
R == CV_8U ? vecBinaryKernel<uchar, uchar, v_float32, float, EwMul, float, v_uint8> :
#endif
R == CV_32F ? vecBinaryKernel<uchar, float, v_float32, float, EwMul> : nullptr;
break;
case CV_8S:
fptr = // scale==1 fast path: whole s8 registers via v_mul_sat
#if CV_SIMD_16F
R == CV_8S ? vecBinaryKernel<schar, schar, v_float16, float, EwMul, float, v_int8> :
#else
R == CV_8S ? vecBinaryKernel<schar, schar, v_float32, float, EwMul, float, v_int8> :
#endif
R == CV_32F ? vecBinaryKernel<schar, float, v_float32, float, EwMul> : nullptr;
break;
case CV_16U:
fptr = // scale==1 fast path: whole u16 registers via v_mul_sat; Wvec=f32 for scale
R == CV_16U ? vecBinaryKernel<ushort, ushort, v_float32, float, EwMul, float, v_uint16> :
R == CV_32F ? vecBinaryKernel<ushort, float, v_float32, float, EwMul> : nullptr;
break;
case CV_16S:
fptr = // scale==1 fast path: whole s16 registers via v_mul_sat; Wvec=f32 for scale
R == CV_16S ? vecBinaryKernel<short, short, v_float32, float, EwMul, float, v_int16> :
R == CV_32F ? vecBinaryKernel<short, float, v_float32, float, EwMul> : nullptr;
break;
case CV_16F:
fptr =
R == CV_16F ? vecBinaryKernel<hfloat, hfloat, v_float32, float, EwMul> :
R == CV_32F ? vecBinaryKernel<hfloat, float, v_float32, float, EwMul> : nullptr;
break;
case CV_16BF:
fptr =
R == CV_16BF ? vecBinaryKernel<bfloat, bfloat, v_float32, float, EwMul> :
R == CV_32F ? vecBinaryKernel<bfloat, float, v_float32, float, EwMul> : nullptr;
break;
case CV_32F:
fptr = R == CV_32F ? vecBinaryKernel<float, float, v_float32, float, EwMul> : nullptr;
break;
case CV_32U:
fptr = // scale==1 fast path (where v_mul_sat32 exists): whole u32 registers, widening
// multiply + saturating narrow; the f64 work vector serves the scale path
#if defined(EW_HAVE_MULSAT32) && CV_SIMD_64F
R == CV_32U ? vecBinaryKernel<unsigned, unsigned, v_float64, double, EwMul, double, v_uint32> :
R == CV_64F ? vecBinaryKernel<unsigned, double, v_float64, double, EwMul> : nullptr;
#elif CV_SIMD_64F
R == CV_32U ? vecBinaryKernel<unsigned, unsigned, v_float64, double, EwMul> :
R == CV_64F ? vecBinaryKernel<unsigned, double, v_float64, double, EwMul> : nullptr;
#else
R == CV_32U ? scalarBinaryKernel<unsigned, unsigned, double, EwMul> :
R == CV_64F ? scalarBinaryKernel<unsigned, double, double, EwMul> : nullptr;
#endif
break;
case CV_32S:
fptr = // scale==1 fast path: see CV_32U
#if defined(EW_HAVE_MULSAT32) && CV_SIMD_64F
R == CV_32S ? vecBinaryKernel<int, int, v_float64, double, EwMul, double, v_int32> :
R == CV_64F ? vecBinaryKernel<int, double, v_float64, double, EwMul> : nullptr;
#elif CV_SIMD_64F
R == CV_32S ? vecBinaryKernel<int, int, v_float64, double, EwMul> :
R == CV_64F ? vecBinaryKernel<int, double, v_float64, double, EwMul> : nullptr;
#else
R == CV_32S ? scalarBinaryKernel<int, int, double, EwMul> :
R == CV_64F ? scalarBinaryKernel<int, double, double, EwMul> : nullptr;
#endif
break;
case CV_64U:
fptr = R == CV_64F ? scalarBinaryKernel<uint64_t, double, double, EwMul> : nullptr;
break;
case CV_64S:
fptr = R == CV_64F ? scalarBinaryKernel<int64_t, double, double, EwMul> : nullptr;
break;
case CV_64F:
#if CV_SIMD_64F
fptr = R == CV_64F ? vecBinaryKernel<double, double, v_float64, double, EwMul> : nullptr;
#else
fptr = R == CV_64F ? scalarBinaryKernel<double, double, double, EwMul> : nullptr;
#endif
break;
default:
;
}
return {fptr, nullptr, 0};
}
// `checked` (decided by the CALLER from the ORIGINAL input types) selects the divide-by-zero
// policy, INDEPENDENT of the work type R: EwDivInt guards /0 -> 0 for integer-semantics division,
// EwDivFlt does not (a/0 -> inf, saturating on a later cast, as cv::divide does for float inputs).
// It must apply on EVERY row - two wide integers (e.g. 32U / 64S) promote to a 64F work type yet
// still need the integer guard, so the float-work rows can't hardcode EwDivFlt.
TKernel getDivFunc_(int T, int R, bool checked)
{
KernelFunc fptr = nullptr;
// SIMD via vecBinaryKernel: div scales the numerator in vec (a*scale/b), EwDivFlt (float a/0 -> inf,
// saturates on cast) / EwDivInt (v_select guards b==0 -> 0 post-facto). Prefer a DIRECT T->T kernel
// (1 pass, saturate on store) - emitBinary probes result==T first; a T->f32/f64 kernel serves an
// explicit float dtype. Work type: f16 for 8-bit (CV_SIMD_16F), else f32; f64 for 32/64-bit.
#define DIV(T_, Tr_, Wv_, W_) (checked ? vecBinaryKernel<T_, Tr_, Wv_, W_, EwDivInt> \
: vecBinaryKernel<T_, Tr_, Wv_, W_, EwDivFlt>)
#if CV_SIMD_16F
#define DIV8(T_) DIV(T_, T_, v_float16, float) // 8-bit T->T, f16 work
#else
#define DIV8(T_) DIV(T_, T_, v_float32, float)
#endif
#if CV_SIMD_64F
#define DIVW(T_, Tr_) DIV(T_, Tr_, v_float64, double) // 32/64-bit, f64 SIMD
#else
#define DIVW(T_, Tr_) (checked ? scalarBinaryKernel<T_, Tr_, double, EwDivInt> \
: scalarBinaryKernel<T_, Tr_, double, EwDivFlt>)
#endif
switch (T)
{
case CV_8U: fptr = R==CV_8U ? DIV8(uchar) : R==CV_32F ? DIV(uchar, float, v_float32, float) : nullptr; break;
case CV_8S: fptr = R==CV_8S ? DIV8(schar) : R==CV_32F ? DIV(schar, float, v_float32, float) : nullptr; break;
case CV_16U: fptr = R==CV_16U ? DIV(ushort, ushort, v_float32, float) : R==CV_32F ? DIV(ushort, float, v_float32, float) : nullptr; break;
case CV_16S: fptr = R==CV_16S ? DIV(short, short, v_float32, float) : R==CV_32F ? DIV(short, float, v_float32, float) : nullptr; break;
case CV_16F: fptr = R==CV_16F ? DIV(hfloat, hfloat, v_float32, float) : R==CV_32F ? DIV(hfloat, float, v_float32, float) : nullptr; break;
case CV_16BF: fptr = R==CV_16BF? DIV(bfloat, bfloat, v_float32, float) : R==CV_32F ? DIV(bfloat, float, v_float32, float) : nullptr; break;
case CV_32F: fptr = R==CV_32F ? DIV(float, float, v_float32, float) : nullptr; break;
case CV_32U: fptr = R==CV_32U ? DIVW(unsigned, unsigned) : R==CV_64F ? DIVW(unsigned, double) : nullptr; break;
case CV_32S: fptr = R==CV_32S ? DIVW(int, int) : R==CV_64F ? DIVW(int, double) : nullptr; break;
case CV_64U: fptr = R==CV_64F ? DIVW(uint64_t, double) : nullptr; break;
case CV_64S: fptr = R==CV_64F ? DIVW(int64_t, double) : nullptr; break;
case CV_64F: fptr = R==CV_64F ? DIVW(double, double) : nullptr; break;
default: ;
}
#undef DIV
#undef DIV8
#undef DIVW
return {fptr, nullptr, 0};
}
// min / max: T x T -> T for every depth. Native v_min/v_max on the matching lane type (8/16/32-bit
// ints, f16/bf16/f32); 64-bit ints and f64 use the scalar path. Op = EwMin or EwMax.
template<class Op>
static TKernel getMinMaxFunc(int T)
{
KernelFunc fptr = nullptr;
switch (T)
{
case CV_8U: fptr = vecBinaryKernel<uchar, uchar, v_uint8, short, Op, uchar>; break;
case CV_8S: fptr = vecBinaryKernel<schar, schar, v_int8, short, Op, schar>; break;
case CV_16U: fptr = vecBinaryKernel<ushort, ushort, v_uint16, int, Op, ushort>; break;
case CV_16S: fptr = vecBinaryKernel<short, short, v_int16, int, Op, short>; break;
case CV_32U: fptr = vecBinaryKernel<unsigned, unsigned, v_uint32, int64_t, Op, unsigned>; break;
case CV_32S: fptr = vecBinaryKernel<int, int, v_int32, int64_t, Op, int>; break;
case CV_16F:
#if CV_SIMD_16F
fptr = vecBinaryKernel<hfloat, hfloat, v_float16, float, Op, hfloat>;
#else
fptr = vecBinaryKernel<hfloat, hfloat, v_float32, float, Op, float>;
#endif
break;
case CV_16BF: fptr = vecBinaryKernel<bfloat, bfloat, v_float32, float, Op, float>; break;
case CV_32F: fptr = vecBinaryKernel<float, float, v_float32, float, Op, float>; break;
case CV_64U: fptr = scalarBinaryKernel<uint64_t, uint64_t, uint64_t, Op>; break;
case CV_64S: fptr = scalarBinaryKernel<int64_t, int64_t, int64_t, Op>; break;
#if CV_SIMD_64F
case CV_64F: fptr = vecBinaryKernel<double, double, v_float64, double, Op, double>; break;
#else
case CV_64F: fptr = scalarBinaryKernel<double, double, double, Op>; break;
#endif
default: ;
}
return {fptr, nullptr, 0};
}
// absdiff: |a-b|, T x T -> R.
TKernel getAbsdiffFunc_(int T, int R)
{
// Signed T has TWO outputs: R==T -> EwAbsdiffS (saturating |a-b| kept in the signed range, native work,
// ONE pass - the depth cv::absdiff keeps); R==(the unsigned type of the same width) -> EwAbsdiff (the
// true |a-b| via v_absdiff, for an explicit unsigned dst). Unsigned/float T only produce R==T. Each case
// returns nullptr for a wrong R (no separate rdepth guard).
KernelFunc fptr = nullptr;
switch (T)
{
case CV_8U: fptr = R == CV_8U ? vecBinaryKernel<uchar, uchar, v_uint8, short, EwAbsdiff, uchar> : nullptr; break;
case CV_16U: fptr = R == CV_16U ? vecBinaryKernel<ushort, ushort, v_uint16, int, EwAbsdiff, ushort> : nullptr; break;
case CV_32U: fptr = R == CV_32U ? vecBinaryKernel<unsigned, unsigned, v_uint32, int64_t, EwAbsdiff, unsigned> : nullptr; break;
case CV_8S: fptr = R == CV_8S ? vecBinaryKernel<schar, schar, v_int8, short, EwAbsdiffS, schar> :
R == CV_8U ? vecBinaryKernel<schar, uchar, v_int8, short, EwAbsdiff, schar> : nullptr; break;
case CV_16S: fptr = R == CV_16S ? vecBinaryKernel<short, short, v_int16, int, EwAbsdiffS, short> :
R == CV_16U ? vecBinaryKernel<short, ushort, v_int16, int, EwAbsdiff, short> : nullptr; break;
case CV_32S: fptr = R == CV_32S ? vecBinaryKernel<int, int, v_int32, int64_t, EwAbsdiffS, int> :
R == CV_32U ? vecBinaryKernel<int, unsigned, v_int32, int64_t, EwAbsdiff, int> : nullptr; break;
#if CV_SIMD_16F
case CV_16F: fptr = R == CV_16F ? vecBinaryKernel<hfloat, hfloat, v_float16, float, EwAbsdiff, hfloat> : nullptr; break;
#else
case CV_16F: fptr = R == CV_16F ? vecBinaryKernel<hfloat, hfloat, v_float32, float, EwAbsdiff, float> : nullptr; break;
#endif
case CV_16BF: fptr = R == CV_16BF ? vecBinaryKernel<bfloat, bfloat, v_float32, float, EwAbsdiff, float> : nullptr; break;
case CV_32F: fptr = R == CV_32F ? vecBinaryKernel<float, float, v_float32, float, EwAbsdiff, float> : nullptr; break;
case CV_64U: fptr = R == CV_64U ? scalarBinaryKernel<uint64_t, uint64_t, uint64_t, EwAbsdiff> : nullptr; break;
case CV_64S: fptr = R == CV_64U ? scalarBinaryKernel<int64_t, uint64_t, int64_t, EwAbsdiff> : nullptr; break;
#if CV_SIMD_64F
case CV_64F: fptr = R == CV_64F ? vecBinaryKernel<double, double, v_float64, double, EwAbsdiff, double> : nullptr; break;
#else
case CV_64F: fptr = R == CV_64F ? scalarBinaryKernel<double, double, double, EwAbsdiff> : nullptr; break;
#endif
default: ;
}
return {fptr, nullptr, 0};
}
// compare: T x T -> u8 mask. Directly-comparable depths (u8/s8/u16/s16/u32/s32/f32) take the SIMD
// vecCompareKernel; f16/bf16 and 64-bit depths fall back to scalarCompareKernel. The returned kernel
// defaults to a 255 mask in TKernel::flags (cv::compare-compatible); kernel.flags=1 gives a 0/1 mask.
template<class Cmp>
static KernelFunc compareByType(int T)
{
switch (T)
{
case CV_8U: return vecCompareKernel<uchar, v_uint8, v_uint8, Cmp>;
case CV_8S: return vecCompareKernel<schar, v_int8, v_uint8, Cmp>;
case CV_16U: return vecCompareKernel<ushort, v_uint16, v_uint16, Cmp>;
case CV_16S: return vecCompareKernel<short, v_int16, v_uint16, Cmp>;
case CV_32U: return vecCompareKernel<unsigned, v_uint32, v_uint32, Cmp>;
case CV_32S: return vecCompareKernel<int, v_int32, v_uint32, Cmp>;
case CV_32F: return vecCompareKernel<float, v_float32, v_uint32, Cmp>;
#if CV_SIMD_16F
case CV_16F: return vecCompareKernel<hfloat, v_float16, v_uint16, Cmp>;
#else
case CV_16F: return vecCompareKernel<hfloat, v_float32, v_uint32, Cmp>; // widen f16->f32
#endif
case CV_16BF: return vecCompareKernel<bfloat, v_float32, v_uint32, Cmp>; // widen bf16->f32 (no native)
case CV_64U: return scalarCompareKernel<uint64_t, uint64_t, Cmp>;
case CV_64S: return scalarCompareKernel<int64_t, int64_t, Cmp>;
#if CV_SIMD_64F
case CV_64F: return vecCompareKernel<double, v_float64, v_uint64, Cmp>;
#else
case CV_64F: return scalarCompareKernel<double, double, Cmp>;
#endif
default: return nullptr;
}
}
TKernel getCmpFunc_(TOp op, int T)
{
// Only 4 physical kernels (eq/ne/gt/ge): LT/LE reuse GT/GE with the operands swapped
// (a<b == b>a, a<=b == b>=a) via the EW_KERNEL_SWAP01 flag, honored by the executor.
KernelFunc f = nullptr;
int flags = 0; // mask value: 0 flag bits => 0/255 (cv::compare); EW_KERNEL_MASK1 => 0/1
switch (op)
{
case OP_CMP_EQ: f = compareByType<EwCmpEq>(T); break;
case OP_CMP_NE: f = compareByType<EwCmpNe>(T); break;
case OP_CMP_GT: f = compareByType<EwCmpGt>(T); break;
case OP_CMP_GE: f = compareByType<EwCmpGe>(T); break;
case OP_CMP_LT: f = compareByType<EwCmpGt>(T); flags = EW_KERNEL_SWAP01; break;
case OP_CMP_LE: f = compareByType<EwCmpGe>(T); flags = EW_KERNEL_SWAP01; break;
default: ;
}
return {f, nullptr, flags};
}
// ===========================================================================
// OP_AND / OP_OR / OP_XOR / OP_NOT: bitwise, type-agnostic (by element size)
// ===========================================================================
// A bit-pattern op ignores the operand's semantic type, so we run it on the UNSIGNED integer whose
// width matches the element (1/2/4/8 bytes). One functor set (EwAnd/EwOr/EwXor) times four widths
// covers every depth; the dispatchers below pick by element size. AND/OR/XOR reuse vecBinaryKernel's
// native same-type path (as min/max do); 64-bit falls to the scalar kernel. NOT is unary.
// bitwise NOT: ~x. Single operand -> always a full contiguous array (no broadcast), so just a flat
// per-row complement. SIMD for 1/2/4-byte elements; 8-byte uses the scalar tail (Vvec unused there).
template<typename T, typename Vvec>
static int notKernel(const void* src0_, size_t s0y, size_t s0x,
const void*, size_t, size_t, const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s0y /= sizeof(T);
dsty /= sizeof(T);
CV_Assert(s0x == 1u || width == 1);
const T* src0 = (const T*)src0_;
T* dst = (T*)dst_;
if (height > 1 && dsty == (size_t)width && s0y == (size_t)width) { width *= height; height = 1; }
for (int y = 0; y < height; y++, src0 += s0y, dst += dsty)
{
int x = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
if constexpr (sizeof(T) <= 4)
{
const int VECSZ = VTraits<Vvec>::vlanes();
for (; x <= width - VECSZ; x += VECSZ)
v_store(dst + x, v_not(vx_load(src0 + x)));
}
#endif
for (; x < width; x++) dst[x] = (T)~src0[x];
}
return 0;
}
template<class Op>
static KernelFunc bitwiseByEsz(int esz)
{
switch (esz)
{
case 1: return vecBinaryKernel<uint8_t, uint8_t, v_uint8, uint8_t, Op, uint8_t>;
case 2: return vecBinaryKernel<uint16_t, uint16_t, v_uint16, uint16_t, Op, uint16_t>;
case 4: return vecBinaryKernel<uint32_t, uint32_t, v_uint32, uint32_t, Op, uint32_t>;
case 8: return scalarBinaryKernel<uint64_t, uint64_t, uint64_t, Op>;
default: return nullptr;
}
}
TKernel getBitwiseFunc_(TOp op, int esz)
{
KernelFunc f = nullptr;
switch (op)
{
case OP_AND: f = bitwiseByEsz<EwAnd>(esz); break;
case OP_OR: f = bitwiseByEsz<EwOr >(esz); break;
case OP_XOR: f = bitwiseByEsz<EwXor>(esz); break;
default: ;
}
return {f, nullptr, 0};
}
TKernel getNotFunc_(int esz)
{
KernelFunc f = nullptr;
switch (esz)
{
case 1: f = notKernel<uint8_t, v_uint8 >; break;
case 2: f = notKernel<uint16_t, v_uint16>; break;
case 4: f = notKernel<uint32_t, v_uint32>; break;
case 8: f = notKernel<uint64_t, v_uint32>; break; // SIMD path compiled out for 8-byte -> scalar ~
default: ;
}
return {f, nullptr, 0};
}
// ===========================================================================
// OP_ADDW (addWeighted): dst = a*alpha + b*beta + gamma, params[0..2] = {alpha, beta, gamma}. Two fused
// v_fma in the work type Wvec - f32 SIMD for u8/s8/u16/s16/f16/bf16/f32; the 32-bit-int/64-bit group
// works in f64 (v_float64 SIMD under CV_SIMD_64F, else use_simd=false scalar). Like vecBinaryKernel but
// WITHOUT its multi-channel short-row
// branch: addWeighted takes plain scalar coefficients (a multi-channel scalar is not optimized, matching
// the classic function). The broadcast branches fold the constant operand's contribution once.
// ===========================================================================
template<typename T, typename Tr, typename Wvec, typename WT, bool use_simd>
static int addWeightedKernel(const void* src0_, size_t s0y, size_t s0x,
const void* src1_, size_t s1y, size_t s1x,
const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double* params, int, void*)
{
s0y /= sizeof(T); s1y /= sizeof(T); dsty /= sizeof(Tr);
CV_Assert((s0x|s1x) == 1u || (s0x|s1x) + (size_t)width == 1u);
const T* src0 = (const T*)src0_;
const T* src1 = (const T*)src1_;
Tr* dst = (Tr*)dst_;
const WT alpha = (WT)params[0], beta = (WT)params[1], gamma = (WT)params[2];
EW_TRY_COLLAPSE(2);
#if (CV_SIMD || CV_SIMD_SCALABLE)
Wvec va{}, vb{}, vg{};
const int VECSZ = VTraits<Wvec>::vlanes();
const bool tail = width >= VECSZ*4 && src0_ != dst_ && src1_ != dst_;
if constexpr (use_simd) {
WT fa=alpha, fb=beta, fg=gamma;
vx_setall_as(&fa, va); vx_setall_as(&fb, vb); vx_setall_as(&fg, vg);
}
#endif
for (int y = 0; y < height; y++, src0 += s0y, src1 += s1y, dst += dsty)
{
int x = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
if constexpr (use_simd)
{
Wvec a0,a1,a2,a3,b0,b1,b2,b3;
if (s0x == s1x) { // both arrays contiguous
for (; x < width; x += VECSZ*4) {
if (x+VECSZ*4 > width) { if (!tail) break; x = width-VECSZ*4; }
vx_load_pair_as(src0+x, a0, a1); vx_load_pair_as(src0+x+VECSZ*2, a2, a3);
vx_load_pair_as(src1+x, b0, b1); vx_load_pair_as(src1+x+VECSZ*2, b2, b3);
a0=v_fma(a0,va,v_fma(b0,vb,vg)); a1=v_fma(a1,va,v_fma(b1,vb,vg));
a2=v_fma(a2,va,v_fma(b2,vb,vg)); a3=v_fma(a3,va,v_fma(b3,vb,vg));
v_store_pair_as(dst+x, a0, a1); v_store_pair_as(dst+x+VECSZ*2, a2, a3);
}
}
else if (s1x == 0) { // src1 broadcast: b*beta+gamma is constant
Wvec bb; vx_setall_as(src1, bb); Wvec bc = v_fma(bb, vb, vg);
for (; x < width; x += VECSZ*4) {
if (x+VECSZ*4 > width) { if (!tail) break; x = width-VECSZ*4; }
vx_load_pair_as(src0+x, a0, a1); vx_load_pair_as(src0+x+VECSZ*2, a2, a3);
a0=v_fma(a0,va,bc); a1=v_fma(a1,va,bc); a2=v_fma(a2,va,bc); a3=v_fma(a3,va,bc);
v_store_pair_as(dst+x, a0, a1); v_store_pair_as(dst+x+VECSZ*2, a2, a3);
}
}
else { // src0 broadcast: a*alpha+gamma is constant
Wvec aa; vx_setall_as(src0, aa); Wvec acg = v_fma(aa, va, vg);
for (; x < width; x += VECSZ*4) {
if (x+VECSZ*4 > width) { if (!tail) break; x = width-VECSZ*4; }
vx_load_pair_as(src1+x, b0, b1); vx_load_pair_as(src1+x+VECSZ*2, b2, b3);
b0=v_fma(b0,vb,acg); b1=v_fma(b1,vb,acg); b2=v_fma(b2,vb,acg); b3=v_fma(b3,vb,acg);
v_store_pair_as(dst+x, b0, b1); v_store_pair_as(dst+x+VECSZ*2, b2, b3);
}
}
}
#endif
if (s0x == s1x) {
for (; x < width; x++) dst[x] = saturate_cast<Tr>((WT)src0[x]*alpha + (WT)src1[x]*beta + gamma);
} else if (s0x == 0) {
const WT ac = (WT)src0[0]*alpha + gamma;
for (; x < width; x++) dst[x] = saturate_cast<Tr>((WT)src1[x]*beta + ac);
} else {
const WT bc = (WT)src1[0]*beta + gamma;
for (; x < width; x++) dst[x] = saturate_cast<Tr>((WT)src0[x]*alpha + bc);
}
}
return 0;
}
TKernel getAddWeightedFunc_(int T, int R)
{
KernelFunc f = nullptr;
#define AWS(Tt, Rr) addWeightedKernel<Tt, Rr, v_float32, float, true>
#if CV_SIMD_16F
#define AWH(Tt) addWeightedKernel<Tt, Tt, v_float16, float, true> // u8/s8 -> same, f16 work
#else
#define AWH(Tt) AWS(Tt, Tt) // no f16: f32 work
#endif
#if CV_SIMD_64F
#define AWD(Tt, Rr) addWeightedKernel<Tt, Rr, v_float64, double, true> // f64 SIMD
#else
#define AWD(Tt, Rr) addWeightedKernel<Tt, Rr, v_float32, double, false> // scalar (v_float32 unused)
#endif
switch (T)
{
case CV_8U: f = R==CV_8U ? AWH(uint8_t) : R==CV_32F ? AWS(uint8_t, float) : nullptr; break;
case CV_8S: f = R==CV_8S ? AWH(int8_t) : R==CV_32F ? AWS(int8_t, float) : nullptr; break;
case CV_16U: f = R==CV_16U ? AWS(uint16_t, uint16_t) : R==CV_32F ? AWS(uint16_t, float) : nullptr; break;
case CV_16S: f = R==CV_16S ? AWS(int16_t, int16_t) : R==CV_32F ? AWS(int16_t, float) : nullptr; break;
case CV_16F: f = R==CV_16F ? AWS(hfloat, hfloat) : R==CV_32F ? AWS(hfloat, float) : nullptr; break;
case CV_16BF:f = R==CV_32F ? AWS(bfloat, float) : nullptr; break;
case CV_32F: f = R==CV_32F ? AWS(float, float) : nullptr; break;
case CV_32U: f = R==CV_32U ? AWD(unsigned, unsigned) : R==CV_64F ? AWD(unsigned, double) : nullptr; break;
case CV_32S: f = R==CV_32S ? AWD(int, int) : R==CV_64F ? AWD(int, double) : nullptr; break;
case CV_64U: f = R==CV_64F ? AWD(uint64_t, double) : nullptr; break;
case CV_64S: f = R==CV_64F ? AWD(int64_t, double) : nullptr; break;
case CV_64F: f = R==CV_64F ? AWD(double, double) : nullptr; break;
default: ;
}
#undef AWS
#undef AWH
#undef AWD
return {f, nullptr, 0};
}
// ===========================================================================
// OP_SELECT: dst = (mask != 0) ? a : b
// ===========================================================================
// The one masking primitive of the engine. It serves both the public texpr select() and the
// masked-op tail: `cv::add(..., mask)` computes the full result into a temp `r`, then a final
// select(mask, r, dst) -> dst lands the masked subset in the (pre-existing) output and PRESERVES
// the rest - dst rides as both an input and the result. That aliasing is safe even under the
// right-edge tail backoff: re-running select over already-blended elements is IDEMPOTENT
// (mask!=0 lanes stay a, mask==0 lanes stay b/dst). Only dst == mask would break (the store
// rewrites the mask before the backoff re-reads it) - that combination falls to the scalar tail.
//
// The mask is one byte per element (bool/u8/s8 - never parameterized by its depth, we just test
// the byte != 0); a/b/dst share one depth, the kernel is templated by the element SIZE only.
// Channels fall out of the broadcast machinery: single-channel data arrives as a (width x 1) tile
// with a per-element mask (smx == 1); n-channel data as a tall-thin tile - channel axis = width,
// the 1-channel mask broadcasting across it (smx == 0, one mask byte per row) - handled by the
// interleaved fast path for 2..4 channels, per-row otherwise. Branches may broadcast (s1x/s2x == 0).
#if (CV_SIMD || CV_SIMD_SCALABLE)
// VECSZ(T) mask bytes -> T-width lanes (u8 direct, u16/u32 via expand)
static inline v_uint8 loadSelectMask(const uchar* m, const v_uint8&) { return vx_load(m); }
static inline v_uint16 loadSelectMask(const uchar* m, const v_uint16&) { return vx_load_expand(m); }
static inline v_uint32 loadSelectMask(const uchar* m, const v_uint32&) { return vx_load_expand_q(m); }
static inline void setallSelect(const uint8_t* p, v_uint8& a) { a = vx_setall_u8(*p); }
static inline void setallSelect(const uint16_t* p, v_uint16& a) { a = vx_setall_u16(*p); }
static inline void setallSelect(const uint32_t* p, v_uint32& a) { a = vx_setall_u32(*p); }
#endif
#ifndef ARITHM_DISPATCHING_ONLY
CV_CPU_OPTIMIZATION_NAMESPACE_END
template<typename T, typename Tvec>
static int selectKernel(const void* mask_, size_t smy, size_t smx,
const void* src1_, size_t s1y, size_t s1x,
const void* src2_, size_t s2y, size_t s2x,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s1y /= sizeof(T); // the 1-byte mask's smy stays in bytes == elements
s2y /= sizeof(T);
dsty /= sizeof(T);
CV_Assert(smx <= 1u && s1x <= 1u && s2x <= 1u);
const uchar* mask = (const uchar*)mask_;
const T* src1 = (const T*)src1_;
const T* src2 = (const T*)src2_;
T* dst = (T*)dst_;
if (height > 1 && dsty == (size_t)width && smy == smx*(size_t)width &&
s1y == s1x*(size_t)width && s2y == s2x*(size_t)width)
{ width *= height; height = 1; }
int y = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
if constexpr (sizeof(T) <= 4)
{
const int VECSZ = VTraits<Tvec>::vlanes();
const Tvec z = v_setzero_<Tvec>();
// n-channel data under a per-pixel mask (the masked-op shape): channel axis = width (2..4),
// one mask byte per row. Process VECSZ rows per iteration - expand the mask once and
// interleave it across the channel lanes.
if (height > VECSZ && 2 <= width && width <= 4 && smx == 0u && smy == 1u &&
s1x == 1u && s1y == (size_t)width && s2x == 1u && s2y == (size_t)width &&
dsty == (size_t)width)
{
constexpr int MAXVECSZ = VTraits<Tvec>::max_nlanes;
T maskbuf[MAXVECSZ*4] = {}; // {} for -Wmaybe-uninitialized only (fully written)
const int dy = VECSZ;
for (; y + dy <= height; y += dy, mask += dy, src1 += width*dy, src2 += width*dy,
dst += width*dy)
{
Tvec m0 = v_ne(loadSelectMask(mask, z), z), m1, m2, m3;
if (width == 2) v_store_interleave(maskbuf, m0, m0);
else if (width == 3) v_store_interleave(maskbuf, m0, m0, m0);
else v_store_interleave(maskbuf, m0, m0, m0, m0);
m0 = vx_load(maskbuf);
m1 = vx_load(maskbuf + VECSZ);
if (width > 2) m2 = vx_load(maskbuf + VECSZ*2);
if (width > 3) m3 = vx_load(maskbuf + VECSZ*3);
v_store(dst, v_select(m0, vx_load(src1), vx_load(src2)));
v_store(dst + VECSZ, v_select(m1, vx_load(src1 + VECSZ), vx_load(src2 + VECSZ)));
if (width > 2)
v_store(dst + VECSZ*2,
v_select(m2, vx_load(src1 + VECSZ*2), vx_load(src2 + VECSZ*2)));
if (width > 3)
v_store(dst + VECSZ*3,
v_select(m3, vx_load(src1 + VECSZ*3), vx_load(src2 + VECSZ*3)));
}
// the remaining < VECSZ rows fall through to the per-row path below
}
else if (smx == 1) // per-element mask - the common case
{
// a branch aliasing dst is fine under the backoff (idempotent select); dst == mask is not
const bool use_tail_trick = width >= VECSZ*2 && dst_ != mask_;
Tvec a, b;
if (s1x == 0) setallSelect(src1, a);
if (s2x == 0) setallSelect(src2, b);
for (; y < height; y++, mask += smy, src1 += s1y, src2 += s2y, dst += dsty)
{
int x = 0;
for (; x < width; x += VECSZ)
{
if (x + VECSZ > width) { if (!use_tail_trick || x == 0) break; x = width - VECSZ; }
Tvec m = v_ne(loadSelectMask(mask + x, z), z);
if (s1x) a = vx_load(src1 + x);
if (s2x) b = vx_load(src2 + x);
v_store(dst + x, v_select(m, a, b));
}
for (; x < width; x++)
dst[x] = mask[x] != 0 ? src1[x*s1x] : src2[x*s2x];
}
return 0;
}
}
#endif
for (; y < height; y++, mask += smy, src1 += s1y, src2 += s2y, dst += dsty)
{
if (smx == 0) // one mask byte per row (n-channel data / broadcast mask)
{
const T* s = mask[0] != 0 ? src1 : src2;
const size_t sx = mask[0] != 0 ? s1x : s2x;
if ((const void*)s != (const void*)dst) // row select from dst itself is a no-op
for (int x = 0; x < width; x++) dst[x] = s[x*sx];
}
else
for (int x = 0; x < width; x++)
dst[x] = mask[x] != 0 ? src1[x*s1x] : src2[x*s2x];
}
return 0;
}
TKernel getSelectFunc_(int mdepth, int T)
{
if (CV_ELEM_SIZE1(mdepth) != 1) // the mask must be a 1-byte type (u8/s8/bool)
return {};
KernelFunc fptr = nullptr;
switch (CV_ELEM_SIZE1(T))
{
case 1: fptr = selectKernel<uint8_t, v_uint8 >; break;
case 2: fptr = selectKernel<uint16_t, v_uint16>; break;
case 4: fptr = selectKernel<uint32_t, v_uint32>; break;
case 8: fptr = selectKernel<uint64_t, v_uint32>; break; // SIMD path compiled out -> scalar
default: ;
}
return {fptr, nullptr, 0};
}
// ===========================================================================
// OP_CLAMP: dst = min(max(x, lo), hi)
// ===========================================================================
// All four operands share one depth (emitTernary unifies them); lo/hi are usually broadcast
// scalars (clamp(img, 10, 200) - literals ride as 0-dim consts with stepx == 0) but may be full
// arrays. clamp is IDEMPOTENT, so the right-edge tail backoff stays on when dst aliases x; only
// dst aliasing lo/hi suppresses it (the store would corrupt the bounds before the re-read).
// NaN note: v_min/v_max lane behavior on NaN is ISA-specific, matching the scalar std::min/max
// unspecifiedness - clamp of NaN is not a contract either way.
#if (CV_SIMD || CV_SIMD_SCALABLE)
static inline void setallClamp(const uchar* p, v_uint8& a) { a = vx_setall_u8(*p); }
static inline void setallClamp(const schar* p, v_int8& a) { a = vx_setall_s8(*p); }
static inline void setallClamp(const ushort* p, v_uint16& a) { a = vx_setall_u16(*p); }
static inline void setallClamp(const short* p, v_int16& a) { a = vx_setall_s16(*p); }
static inline void setallClamp(const unsigned* p, v_uint32& a) { a = vx_setall_u32(*p); }
static inline void setallClamp(const int* p, v_int32& a) { a = vx_setall_s32(*p); }
static inline void setallClamp(const float* p, v_float32& a) { a = vx_setall_f32(*p); }
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
static inline void setallClamp(const double* p, v_float64& a) { a = vx_setall_f64(*p); }
#endif
#endif
}} // cv::hal::
template<typename T, typename Tvec>
static int clampKernel(const void* src0_, size_t s0y, size_t s0x,
const void* lo_, size_t s1y, size_t s1x,
const void* hi_, size_t s2y, size_t s2x,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s0y /= sizeof(T);
s1y /= sizeof(T);
s2y /= sizeof(T);
dsty /= sizeof(T);
CV_Assert(s0x <= 1u && s1x <= 1u && s2x <= 1u);
const T* src0 = (const T*)src0_;
const T* lo = (const T*)lo_;
const T* hi = (const T*)hi_;
T* dst = (T*)dst_;
if (height > 1 && dsty == (size_t)width && s0y == s0x*(size_t)width &&
s1y == s1x*(size_t)width && s2y == s2x*(size_t)width)
{ width *= height; height = 1; }
for (int y = 0; y < height; y++, src0 += s0y, lo += s1y, hi += s2y, dst += dsty)
{
int x = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
if (s0x == 1)
{
const int VECSZ = VTraits<Tvec>::vlanes();
const bool use_tail_trick = width >= VECSZ*2 && lo_ != dst_ && hi_ != dst_;
Tvec vlo, vhi;
if (s1x == 0) setallClamp(lo, vlo);
if (s2x == 0) setallClamp(hi, vhi);
for (; x < width; x += VECSZ)
{
if (x + VECSZ > width) { if (!use_tail_trick || x == 0) break; x = width - VECSZ; }
if (s1x) vlo = vx_load(lo + x);
if (s2x) vhi = vx_load(hi + x);
v_store(dst + x, v_min(v_max(vx_load(src0 + x), vlo), vhi));
}
}
#endif
for (; x < width; x++)
{
T v = src0[x*s0x], l = lo[x*s1x], h = hi[x*s2x];
dst[x] = std::min(std::max(v, l), h);
}
}
return 0;
}
// Scalar-only clamp for the depths without a native SIMD lane type here (f16/bf16 - compared in
// float; 64-bit ints).
template<typename T, typename WT>
static int scalarClampKernel(const void* src0_, size_t s0y, size_t s0x,
const void* lo_, size_t s1y, size_t s1x,
const void* hi_, size_t s2y, size_t s2x,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s0y /= sizeof(T); s1y /= sizeof(T); s2y /= sizeof(T); dsty /= sizeof(T);
CV_Assert(s0x <= 1u && s1x <= 1u && s2x <= 1u);
const T* src0 = (const T*)src0_;
const T* lo = (const T*)lo_;
const T* hi = (const T*)hi_;
T* dst = (T*)dst_;
if (height > 1 && dsty == (size_t)width && s0y == s0x*(size_t)width &&
s1y == s1x*(size_t)width && s2y == s2x*(size_t)width)
{ width *= height; height = 1; }
for (int y = 0; y < height; y++, src0 += s0y, lo += s1y, hi += s2y, dst += dsty)
for (int x = 0; x < width; x++)
{
WT v = (WT)src0[x*s0x], l = (WT)lo[x*s1x], h = (WT)hi[x*s2x];
dst[x] = saturate_cast<T>(std::min(std::max(v, l), h));
}
return 0;
}
TKernel getClampFunc_(int T)
{
KernelFunc fptr = nullptr;
switch (T)
{
case CV_8U: fptr = clampKernel<uchar, v_uint8 >; break;
case CV_8S: fptr = clampKernel<schar, v_int8 >; break;
case CV_16U: fptr = clampKernel<ushort, v_uint16 >; break;
case CV_16S: fptr = clampKernel<short, v_int16 >; break;
case CV_32U: fptr = clampKernel<unsigned, v_uint32 >; break;
case CV_32S: fptr = clampKernel<int, v_int32 >; break;
case CV_32F: fptr = clampKernel<float, v_float32>; break;
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
case CV_64F: fptr = clampKernel<double, v_float64>; break;
#else
case CV_64F: fptr = scalarClampKernel<double, double>; break;
#endif
case CV_16F: fptr = scalarClampKernel<hfloat, float>; break;
case CV_16BF: fptr = scalarClampKernel<bfloat, float>; break;
case CV_64U: fptr = scalarClampKernel<uint64_t, uint64_t>; break;
case CV_64S: fptr = scalarClampKernel<int64_t, int64_t >; break;
default: ;
}
return {fptr, nullptr, 0};
}
static int expandKernel(size_t s0y, size_t s0x,
void* dst_, size_t dsty, int width, int height,
int esz1)
{
uchar* dst = (uchar*)dst_;
if (s0x == 0u && width > 1) {
int h = s0y > 0u ? height : 1;
CV_Assert(esz1 == 1 || esz1 == 2 || esz1 == 4 || esz1 == 8);
for (int y = 0; y < h; y++) {
if (esz1 == 1) {
uchar* rowptr = dst + dsty*y;
uchar val = rowptr[0];
for (int x = 1; x < width; x++) rowptr[x] = val;
}
else if (esz1 == 2) {
ushort* rowptr = (ushort*)(dst + dsty*y);
ushort val = rowptr[0];
for (int x = 1; x < width; x++) rowptr[x] = val;
}
else if (esz1 == 4) {
unsigned* rowptr = (unsigned*)(dst + dsty*y);
unsigned val = rowptr[0];
for (int x = 1; x < width; x++) rowptr[x] = val;
}
else {
uint64_t* rowptr = (uint64_t*)(dst + dsty*y);
uint64_t val = rowptr[0];
for (int x = 1; x < width; x++) rowptr[x] = val;
}
}
}
if (s0y == 0) {
for (int y = 1; y < height; y++)
memcpy(dst + dsty*y, dst, (size_t)esz1*width);
}
return 0;
}
static int castKernel(const void* src0_, size_t s0y, size_t s0x,
const void*, size_t, size_t,
const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double* params, int esz1, void* userdata)
{
BinaryFunc castfunc = (BinaryFunc)userdata;
castfunc((const uchar*)src0_, s0y, nullptr, 0, (uchar*)dst_, dsty,
Size((s0x > 0u ? width : 1), (s0y > 0u ? height : 1)), (void*)params);
return expandKernel(s0y, s0x, dst_, dsty, width, height, esz1);
}
// ===========================================================================
// Per-op entry points (this baseline). The op-level getElemwiseFunc() dispatcher and the regular
// get*Func() forwarders live in arithm.dispatch.cpp.
// ===========================================================================
// OP_CAST / OP_CONVERT_SCALE: wrap core's convert BinaryFunc (carried in kernel.userdata) in
// castKernel, which runs it over the distinct sub-region and then expands across broadcast axes.
TKernel getCastFunc_(int sdepth, int ddepth, bool scaled)
{
BinaryFunc cvt = scaled ? getConvertScaleFunc(sdepth, ddepth) : getConvertFunc(sdepth, ddepth);
return {castKernel, (void*)cvt, CV_ELEM_SIZE1(ddepth)};
}
// Non-template wrappers around the templated kernel selectors (ADD/SUB share getAddSubFunc, MIN/MAX
// share getMinMaxFunc). min/max are T x T -> T, so R is ignored (the dispatcher validates R == T).
TKernel getAddFunc_(int T, int R) { return getAddSubFunc<EwAdd>(T, R); }
TKernel getSubFunc_(int T, int R) { return getAddSubFunc<EwSub>(T, R); }
TKernel getMinFunc_(int T, int R) { (void)R; return getMinMaxFunc<EwMin>(T); }
TKernel getMaxFunc_(int T, int R) { (void)R; return getMinMaxFunc<EwMax>(T); }
TKernel getHypotFunc_(int T, int R)
{
if (R != T)
return {};
KernelFunc fptr = nullptr;
switch (T)
{
case CV_16F: fptr = vecBinaryKernel<hfloat, hfloat, v_float32, float, EwHypot>; break;
case CV_16BF: fptr = vecBinaryKernel<bfloat, bfloat, v_float32, float, EwHypot>; break;
case CV_32F: fptr = vecBinaryKernel<float, float, v_float32, float, EwHypot>; break;
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
case CV_64F: fptr = vecBinaryKernel<double, double, v_float64, double, EwHypot>; break;
#else
case CV_64F: fptr = scalarBinaryKernel<double, double, double, EwHypot>; break;
#endif
default: ;
}
return {fptr, nullptr, 0};
}
TKernel getAtan2Func_(int T, int R)
{
if (R != T)
return {};
KernelFunc fptr = nullptr;
switch (T)
{
case CV_16F: fptr = vecBinaryKernel<hfloat, hfloat, v_float32, float, EwAtan2>; break;
case CV_16BF: fptr = vecBinaryKernel<bfloat, bfloat, v_float32, float, EwAtan2>; break;
case CV_32F: fptr = vecBinaryKernel<float, float, v_float32, float, EwAtan2>; break;
case CV_64F: fptr = scalarBinaryKernel<double, double, double, EwAtan2>; break; // exact std::atan2
default: ;
}
return {fptr, nullptr, 0};
}
#endif // CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
CV_CPU_OPTIMIZATION_NAMESPACE_END
}} // namespace cv::ew
modules/core/src/arithm_expr.cpp
+1827
View File
@@ -0,0 +1,1827 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// The new element-wise expression engine: op metadata, the graph compiler (type inference + cast
// insertion + liveness), the executor (broadcast traversal via BroadcastOp), the hand builders
// (makeBinaryArithProgram/makeAddWeightedProgram) and the cv::expression parser. Merged from the
// prototype's ew_op/ew_compile/ew_exec/ew_parser. Kernels are reached through the dispatchers in
// arithm.dispatch.cpp (getElemwiseFunc/getDivFunc, declared in arithm_expr.hpp).
#include "precomp.hpp"
#include "arithm_expr.hpp"
#include <algorithm>
#include <climits>
#include <cmath>
#include <cstdlib>
#include <cstring>
#include <map>
#include <ostream>
#include <string>
namespace cv { namespace ew {
// ============================ op metadata (was ew_op.cpp) ============================
const char* opName(TOp op)
{
switch (op)
{
case OP_NOP: return "nop";
case OP_NEG: return "neg";
case OP_ABS: return "abs";
case OP_NOT: return "not";
case OP_SQRT: return "sqrt";
case OP_EXP: return "exp";
case OP_LOG: return "log";
case OP_SIN: return "sin";
case OP_COS: return "cos";
case OP_TANH: return "tanh";
case OP_ERF: return "erf";
case OP_RELU: return "relu";
case OP_CAST: return "cast";
case OP_ADD: return "add";
case OP_SUB: return "sub";
case OP_MUL: return "mul";
case OP_DIV: return "div";
case OP_POW: return "pow";
case OP_MIN: return "min";
case OP_MAX: return "max";
case OP_ABSDIFF: return "absdiff";
case OP_HYPOT: return "hypot";
case OP_ATAN2: return "atan2";
case OP_AND: return "and";
case OP_OR: return "or";
case OP_XOR: return "xor";
case OP_CMP_EQ: return "cmp_eq";
case OP_CMP_NE: return "cmp_ne";
case OP_CMP_LT: return "cmp_lt";
case OP_CMP_LE: return "cmp_le";
case OP_CMP_GT: return "cmp_gt";
case OP_CMP_GE: return "cmp_ge";
case OP_CLAMP: return "clamp";
case OP_SELECT: return "select";
case OP_CONVERT_SCALE: return "convert_scale";
default: return "?";
}
}
// --- op category (declared in ew_op.hpp) --------------------------------------------------
ElemwiseCategory opCategory(TOp op)
{
switch (op)
{
case OP_AND: case OP_OR: case OP_XOR: case OP_NOT:
return CAT_BITWISE;
case OP_CMP_EQ: case OP_CMP_NE: case OP_CMP_LT:
case OP_CMP_LE: case OP_CMP_GT: case OP_CMP_GE:
return CAT_COMPARE;
case OP_SQRT: case OP_EXP: case OP_LOG:
case OP_SIN: case OP_COS: case OP_TANH: case OP_ERF: case OP_ATAN2: case OP_RELU:
return CAT_MATH;
case OP_CAST: case OP_CONVERT_SCALE:
return CAT_CAST;
case OP_SELECT:
return CAT_SELECT;
default:
return CAT_ARITH; // add/sub/mul/div/pow/min/max/absdiff/neg/abs/clamp
}
}
// ============================ compiler (was ew_compile.cpp) ============================
// ---------------------------------------------------------------------------
// Type inference helpers (deliberately small so they can grow later).
// ---------------------------------------------------------------------------
static bool isFloatDepth(int d)
{
return d == CV_16F || d == CV_16BF || d == CV_32F || d == CV_64F;
}
// Can `depth` hold `v` exactly? Floats: yes (close enough for our promotion). Integers: only if v
// is integral and in range. Used so a const operand (e.g. 2.5 in a*2.5) is NOT quantized into a
// narrow-integer direct kernel - such an op must fall back to the float working type instead.
static bool depthRepresents(double v, int depth)
{
if (isFloatDepth(depth)) return true;
if (v != std::floor(v)) return false;
switch (depth)
{
case CV_8U: return v >= 0 && v <= 255;
case CV_8S: return v >= -128 && v <= 127;
case CV_16U: return v >= 0 && v <= 65535;
case CV_16S: return v >= -32768 && v <= 32767;
case CV_32U: return v >= 0 && v <= 4294967295.0;
case CV_32S: return v >= -2147483648.0 && v <= 2147483647.0;
case CV_64U: return v >= 0 && v <= 18446744073709551615.0;
case CV_64S: return v >= -9223372036854775808.0 && v <= 9223372036854775807.0;
default: return false;
}
}
// [min, max] representable value of an integer depth, as doubles (exact for <=32-bit; the 64-bit
// endpoints round to the nearest double). Used by the compare-with-const boundary rewrite.
static void intRange(int depth, double& lo, double& hi)
{
switch (depth)
{
case CV_8U: lo = 0; hi = 255; break;
case CV_8S: lo = -128; hi = 127; break;
case CV_16U: lo = 0; hi = 65535; break;
case CV_16S: lo = -32768; hi = 32767; break;
case CV_32U: lo = 0; hi = 4294967295.0; break;
case CV_32S: lo = -2147483648.0; hi = 2147483647.0; break;
case CV_64U: lo = 0; hi = 18446744073709551615.0; break;
case CV_64S: lo = -9223372036854775808.0; hi = 9223372036854775807.0; break;
default: lo = 0; hi = 0; break;
}
}
// numpy-style arithmetic promotion - INTEGER-PRESERVING and COMMUTATIVE. Same signedness -> the wider
// integer (keeps the sign). Mixed sign -> a SIGNED result wide enough to hold the unsigned operand's
// range (8u+8s -> 16s, 16u+16s -> 32s, 32u+32s -> 64s; 64-bit mixed -> 64F, no 128-bit int exists).
// Any float -> the smallest float covering both operands (8-bit int / 16F / 16BF -> a 16-bit float,
// 16-bit int / 32F -> 32F, 32/64-bit int / 64F -> 64F; two DISTINCT 16-floats widen to 32F). A flexible
// operand (EW_DEPTH_NONE == -1) lands in the reserved slot 0 of every LUT (size 0, unsigned, non-float),
// so the integer path just returns the other operand - no explicit NONE guard needed. Declared in
// ew_op.hpp.
//
// Packed lookup tables, one field per depth at slot (depth+1):
// leszlut - 3-bit size class: 8-bit -> 1, 16-bit -> 2, 32-bit -> 3, 64-bit -> 4 (NONE/Bool -> 0)
// signlut - 1 bit, set for signed integer depths
// fltlut - 1 bit, set for float depths
int promoteArith(int a, int b)
{
constexpr uint64_t leszlut = 034412243322110ULL;
constexpr unsigned signlut = 0b1001111110100u;
constexpr unsigned fltlut = 0b1111000000u;
if (a == b) return a;
int wa = int((leszlut >> (a+1)*3) & 7u), wb = int((leszlut >> (b+1)*3) & 7u);
int fa = int((fltlut >> (a+1)) & 1u), fb = int((fltlut >> (b+1)) & 1u);
if (fa + fb == 0) // both integer
{
int sa = int((signlut >> (a+1)) & 1u), sb = int((signlut >> (b+1)) & 1u);
if (sa == sb) // same signedness -> the wider one (keeps the sign)
return wa >= wb ? a : b;
// mixed sign -> a SIGNED result holding the unsigned operand: the signed width if it is already
// wider, else one step past the unsigned width; past 64 bits there is no int -> f64.
int rw = std::max(wa + (1 - sa), wb + (1 - sb));
constexpr int ilut = (CV_64F << 5*5) | (CV_64S << 4*5) | (CV_32S << 3*5) |
(CV_16S << 2*5) | (CV_8S << 1*5) | (CV_8S << 0*5);
return (ilut >> rw*5) & CV_MAT_DEPTH_MASK;
}
// at least one float -> the smallest float covering both operands. Lift each integer operand to the
// float size class that holds it (8-bit -> 16-float, 16-bit -> 32F, 32/64-bit -> 64F); a float
// operand keeps its own. Size 2 with exactly one float -> that float (a 8-bit int + 16-float pair);
// otherwise 32F, or 64F once the size class reaches 4.
wa += 1 - fa;
wb += 1 - fb;
int maxw = std::max(wa, wb);
if (maxw == 2 && fa + fb == 1)
return fa*a + (1 - fa)*b;
return CV_32F + (maxw >= 4);
}
// Declared in ew_op.hpp. A signed integer |a-b| reaches 2^width-1, so absdiff returns the unsigned
// type of the same width; unsigned/float depths are unchanged.
int absdiffResultDepth(int depth)
{
switch (depth)
{
case CV_8S: return CV_8U;
case CV_16S: return CV_16U;
case CV_32S: return CV_32U;
case CV_64S: return CV_64U;
default: return depth;
}
}
// numpy-ish promotion of two KNOWN depths for the COMPUTE type (float dominates; the wider integer
// otherwise, and - unlike promoteArith - two WIDE integers float-promote: promote2(16U,64S)=64F).
// This is cv::add's "wtype": the type both operands are brought to before the op runs.
static int promote2(int a, int b)
{
if (a == b) return a;
constexpr unsigned lbits = 3, lmask = (1u << lbits) - 1u;
const uint64_t typelut = (uint64_t)((0ULL << CV_8U*lbits) | (0ULL << CV_8S*lbits) |
(1ULL << CV_16U*lbits) | (1ULL << CV_16S*lbits) |
(2ULL << CV_32U*lbits) | (2ULL << CV_32S*lbits) |
(3ULL << CV_16F*lbits) | (3ULL << CV_16BF*lbits) |
(3ULL << CV_32F*lbits) | (4ULL << CV_64F*lbits) |
(4ULL << CV_64S*lbits) | (4ULL << CV_64U*lbits));
unsigned pr_a = unsigned((typelut >> (a*lbits)) & lmask);
unsigned pr_b = unsigned((typelut >> (b*lbits)) & lmask);
unsigned max_pr = std::max(pr_a, pr_b);
constexpr unsigned dbits = CV_CN_SHIFT, dmask = (1u << dbits) - 1u;
const unsigned ctypelut = ((CV_16S << 0*dbits) | (CV_32S << 1*dbits) | (CV_64S << 2*dbits) |
(CV_32F << 3*dbits) | (CV_64F << 4*dbits));
return int((ctypelut >> (max_pr*dbits)) & dmask);
}
// Common type for a bit-pattern op (AND/OR/XOR). Unlike promoteArith there is NO numeric promotion:
// a bitwise op keeps the operand's own type, and a scalar operand simply takes the array's type and
// is reinterpreted by its bits. A flexible CONST (EW_DEPTH_NONE) yields to the concrete operand; two
// concrete depths must share the element WIDTH (cv::bitwise requires equal types) - if they differ we
// keep the wider one (the kernel dispatches by element size). NONE x NONE stays NONE (caller defaults).
static int promoteBitwise(int a, int b)
{
if (a == EW_DEPTH_NONE) return b;
if (b == EW_DEPTH_NONE) return a;
if (a == b) return a;
return CV_ELEM_SIZE1(a) >= CV_ELEM_SIZE1(b) ? a : b;
}
// A wide type in which add(depth,depth->wide) exists and the sum is held without a premature clamp
// (the signed-widening fallback for ADD/SUB, whose difference may go negative).
static int safeWide(int depth)
{
constexpr unsigned lbits = 3, lmask = (1u << lbits) - 1u;
const uint64_t typelut = (uint64_t)((0ULL << CV_8U*lbits) | (0ULL << CV_8S*lbits) |
(1ULL << CV_16U*lbits) | (1ULL << CV_16S*lbits) |
(2ULL << CV_32U*lbits) | (2ULL << CV_32S*lbits) |
(3ULL << CV_16F*lbits) | (3ULL << CV_16BF*lbits) |
(3ULL << CV_32F*lbits) | (4ULL << CV_64F*lbits) |
(4ULL << CV_64S*lbits) | (4ULL << CV_64U*lbits));
unsigned pr_depth = unsigned((typelut >> (depth*lbits)) & lmask);
constexpr unsigned dbits = CV_CN_SHIFT, dmask = (1u << dbits) - 1u;
const unsigned ctypelut = ((CV_16S << 0*dbits) | (CV_32S << 1*dbits) | (CV_64S << 2*dbits) |
(CV_32F << 3*dbits) | (CV_64F << 4*dbits));
return int((ctypelut >> (pr_depth*dbits)) & dmask);
}
// Inline capacity for the small per-const double scratch used during type inference; larger
// channel counts (e.g. Vec<_,16>) just spill the AutoBuffer to the heap.
enum { MAX_LOCAL_CN = 16 };
// Append `channels` values of depth `srcdepth` (raw bytes at `data`, or zero-filled if null) to
// constbuf, padded to a whole number of uint64_t slots. Returns the offset (in uint64_t units).
static size_t appendConstBuf(AutoBuffer<uint64_t, 16>& constbuf, int srcdepth, const void* data, int channels)
{
const int cn = std::max(1, channels);
const size_t nbytes = (size_t)cn * CV_ELEM_SIZE1(srcdepth);
const size_t nu64 = (nbytes + sizeof(uint64_t) - 1) / sizeof(uint64_t);
const size_t ofs = constbuf.size();
constbuf.resize(ofs + nu64);
uchar* dst = (uchar*)(constbuf.data() + ofs);
if (data) memcpy(dst, data, nbytes); else memset(dst, 0, nbytes);
return ofs;
}
// Read a CONST slot's source values (constbuf, `srcdepth`) as doubles into `buf`. Returns channels.
static int constDoubles(const TExpr& e, int s, AutoBuffer<double, MAX_LOCAL_CN>& buf)
{
const TExpr::Arg& a = e.arginfo[s];
const int cn = std::max(1, a.channels);
buf.resize(cn);
const uchar* src = (const uchar*)(e.constbuf.data() + a.constofs);
if (a.srcdepth == CV_64F)
memcpy(buf.data(), src, (size_t)cn * sizeof(double)); // common (Scalar / parsed literal)
else
getConvertFunc(a.srcdepth, CV_64F)(src, 0, nullptr, 0, (uchar*)buf.data(), 0, Size(cn, 1), nullptr);
return cn;
}
// Materialize a flexible CONST `s` at depth `d`, or return `s` unchanged for a typed operand
// (the emit* layer then casts it to the compute depth). Used by the emit* policy layers.
static inline bool isFlexConst(const TExpr& e, int s)
{
return e.arginfo[s].kind == TExpr::CONST && e.arginfo[s].depth == EW_DEPTH_NONE;
}
// Can a flexible CONST `s` be represented exactly at depth `d`? (typed operands trivially "fit").
static bool constFits(const TExpr& e, int s, int d)
{
if (!isFlexConst(e, s)) return true;
AutoBuffer<double, MAX_LOCAL_CN> v;
int cn = constDoubles(e, s, v);
for (int ch = 0; ch < cn; ch++)
if (!depthRepresents(v[ch], d)) return false;
return true;
}
// ---------------------------------------------------------------------------
// TExpr::emitBinary(): type policy for a 2-input op. Derive the result depth, the compute depth and a
// wide fallback per op family, then cast both operands and emit (direct, else wide+narrow). Operands
// may be typed (INPUT/TEMP) or a flexible CONST (materialized at the compute depth here).
// ---------------------------------------------------------------------------
int TExpr::emitBinary(TOp op, int a, int b, int rdepth, const Scalar& params)
{
// addWeighted a*alpha + b*beta + gamma (params = {alpha, beta, gamma}): ONE fused kernel (two v_fma).
// Inputs are the same type T (cast to a common type if not). The kernel outputs T/f32 (small ints,
// f16/bf16, f32) or f64 directly; for any other requested rdepth it computes in the work type W and a
// final cast narrows it.
if (op == OP_ADDW)
{
int Tt = arginfo[a].depth;
if (arginfo[a].depth != arginfo[b].depth)
{
Tt = promoteArith(arginfo[a].depth, arginfo[b].depth);
a = maybeAddCast(a, Tt); b = maybeAddCast(b, Tt);
}
if (rdepth == EW_DEPTH_NONE) rdepth = Tt; // default dtype = input depth
TKernel k = getElemwiseFunc(OP_ADDW, Tt, Tt, EW_DEPTH_NONE, rdepth);
int outD = rdepth;
if (!k.fptr) // no direct T->rdepth kernel: compute in W, cast
{
outD = (Tt==CV_32U || Tt==CV_32S || Tt==CV_64U || Tt==CV_64S || Tt==CV_64F || rdepth==CV_64F)
? CV_64F : CV_32F;
k = getElemwiseFunc(OP_ADDW, Tt, Tt, EW_DEPTH_NONE, outD);
}
const int out = addTemp(outD);
addInsn(OP_ADDW, a, b, 0, out, k, Scalar(params[0], params[1], params[2]));
if (outD == rdepth) return out;
const int out2 = addTemp(rdepth);
addInsn(OP_CAST, out, 0, 0, out2);
return out2;
}
const int nd0 = isFlexConst(*this, a) ? EW_DEPTH_NONE : arginfo[a].depth;
const int nd1 = isFlexConst(*this, b) ? EW_DEPTH_NONE : arginfo[b].depth;
const ElemwiseCategory cat = opCategory(op);
// result depth (auto unless forced): compare -> mask (u8); everything else -> promoteArith.
int result = rdepth;
if (result == EW_DEPTH_NONE)
{
result = (cat == CAT_COMPARE) ? CV_8U
: (cat == CAT_BITWISE) ? promoteBitwise(nd0, nd1) // no numeric promotion for bit ops
: promoteArith(nd0, nd1);
if (result == EW_DEPTH_NONE) result = CV_32F; // const (op) const
}
// common type over the CONCRETE operands (a flexible const does not force it).
int base;
if (nd0 == EW_DEPTH_NONE && nd1 == EW_DEPTH_NONE) base = result;
else if (nd0 == EW_DEPTH_NONE) base = nd1;
else if (nd1 == EW_DEPTH_NONE) base = nd0;
else base = promote2(nd0, nd1);
// COMPARE of an INTEGER array against a threshold that doesn't fit that type as-is (fractional,
// out-of-range, or EQ/NE of a non-representable value): emit a SINGLE NATIVE integer compare instead
// of widening both sides to f64. Per channel the relation is either a REAL boundary (a>=B / a<=B) or
// a CONSTANT (always-false / always-true). Boundaries: a>t==a>=floor(t)+1; a>=t==a>=ceil(t);
// a<t==a<=ceil(t)-1; a<=t==a<=floor(t); a==(non-rep) never true; a!=(non-rep) always true. One family
// op (GE for GT/GE, LE for LT/LE, else EQ/NE) runs with a placeholder threshold on const channels;
// the per-channel fix-up result = (rawmask & M) | V is FOLDED into the compare kernel via its flags
// (M=255 real / 0 const, V=255 always-true / 0 else) - no extra pass, no separate patch kernel.
if (cat == CAT_COMPARE && nd0 != EW_DEPTH_NONE && !isFloatDepth(nd0) &&
isFlexConst(*this, b) && !constFits(*this, b, base))
{
double lo, hi; intRange(nd0, lo, hi);
if (hi > lo) // hi<=lo => a depth intRange doesn't model (CV_Bool ...): leave to the f64 path
{
enum { REAL, CFALSE, CTRUE };
AutoBuffer<double, MAX_LOCAL_CN> tv; int cn = constDoubles(*this, b, tv);
CV_Assert(cn <= 4); // a CONST is <= 4 channels (addConst) - fixed arrays, no heap
int kind[4] = {}; double bound[4] = {}; // {} for -Wmaybe-uninitialized only
// (filled for all cn used below)
const TOp fam = (op == OP_CMP_GT || op == OP_CMP_GE) ? OP_CMP_GE
: (op == OP_CMP_LT || op == OP_CMP_LE) ? OP_CMP_LE : op; // EQ/NE unchanged
for (int c = 0; c < cn; c++)
{
const double t = tv[c]; int k; double B = t;
switch (op)
{
case OP_CMP_GT: B = std::floor(t)+1; k = B > hi ? CFALSE : B <= lo ? CTRUE : REAL; break;
case OP_CMP_GE: B = std::ceil(t); k = B > hi ? CFALSE : B <= lo ? CTRUE : REAL; break;
case OP_CMP_LT: B = std::ceil(t)-1; k = B < lo ? CFALSE : B >= hi ? CTRUE : REAL; break;
case OP_CMP_LE: B = std::floor(t); k = B < lo ? CFALSE : B >= hi ? CTRUE : REAL; break;
case OP_CMP_EQ: k = depthRepresents(t, nd0) ? REAL : CFALSE; break; // a==non-rep -> false
default: k = depthRepresents(t, nd0) ? REAL : CTRUE; break; // NE: a!=non-rep -> true
}
kind[c] = k; bound[c] = B;
}
// If every channel lands on ONE op (single-channel, all-real, all-false, all-true, or
// real+true for a GE/LE family) the compare ALONE yields the result - a plain native compare,
// no fix-up. REAL -> fam(bound); forced-TRUE -> the family's always-true form (GE lo / LE hi);
// forced-FALSE -> "a < lo".
auto ucOp = [&](int c){ return kind[c]==REAL ? fam : kind[c]==CFALSE ? OP_CMP_LT
: (fam == OP_CMP_LE ? OP_CMP_LE : OP_CMP_GE); };
auto ucThr = [&](int c){ return kind[c]==REAL ? bound[c]
: (kind[c]==CTRUE && fam==OP_CMP_LE) ? hi : lo; };
const TOp u0 = ucOp(0);
bool uniform = true;
for (int c = 1; c < cn && uniform; c++) uniform = (ucOp(c) == u0);
if (uniform)
{
Scalar s; for (int c = 0; c < cn; c++) s[c] = ucThr(c);
b = addConst(EW_DEPTH_NONE, s, cn); op = u0; // rewrite threshold + op -> fall through
}
else
{
// genuine per-channel op split (only a multi-channel scalar can cause it -> the executor
// lays it out as a short-row tile). One family compare with a placeholder threshold on
// const channels; the per-channel fix-up (rawmask & M) | V is FOLDED into the kernel via
// its flags (M=255 real / 0 const, V=255 always-true / 0 else) - no extra pass.
Scalar ts; int patchFlags = 0;
for (int c = 0; c < cn; c++)
{
ts[c] = kind[c]==REAL ? bound[c] : lo; // in-range placeholder for a const channel
const int mbits = kind[c]==REAL ? 3 : 0; // M: real -> 255 (keep), const -> 0
const int vbits = kind[c]==CTRUE ? 3 : 0; // V: always-true -> 255, else 0
patchFlags |= (mbits | (vbits << 2)) << (EW_CMP_PATCH_SHIFT + c*4);
}
TKernel kern = getElemwiseFunc(fam, base, base, EW_DEPTH_NONE, result);
kern.flags |= EW_CMP_PATCH | patchFlags;
const int out = addTemp(result);
addInsn(fam, a, addConst(base, ts, cn), 0, out, kern);
return out;
}
}
}
// compute depth + wide fallback per family:
// MIN/MAX, AND/OR/XOR : T x T -> T, never widen (depth = result)
// ABSDIFF : T x T -> unsigned same width for signed ints (8s->8u, ...)
// COMPARE : common type -> mask (depth = base)
// MUL/DIV : float work (f64 if wide) (matches cv::multiply/divide)
// POW : float work
// ADD/SUB : common type, signed wide fallback
int depth, wide;
bool divGuard = false;
switch (op)
{
case OP_MIN: case OP_MAX:
case OP_AND: case OP_OR: case OP_XOR:
depth = result; wide = result; break;
case OP_ABSDIFF:
{
// operands meet at the integer-preserving common type; |a-b| of a signed integer needs the
// UNSIGNED type of the same width (8s->8u, ...), which is both the result and the safe type.
int common = (nd0 == EW_DEPTH_NONE) ? nd1
: (nd1 == EW_DEPTH_NONE) ? nd0 : promoteArith(nd0, nd1);
if (common == EW_DEPTH_NONE) common = result; // both operands flexible consts
depth = common;
wide = absdiffResultDepth(common);
if (rdepth == EW_DEPTH_NONE) result = wide; // auto result: unsigned same width
break;
}
case OP_CMP_EQ: case OP_CMP_NE: case OP_CMP_LT:
case OP_CMP_LE: case OP_CMP_GT: case OP_CMP_GE:
// compare in the common operand type; if a flexible-const threshold does not FIT that type
// (out of range, e.g. u8 > -10, or fractional, e.g. u8 > 2.5) fall back to f64 so it is NOT
// saturated into the operand type (which would move the boundary). f64 is exact for every
// integer array up to 32-bit; only a 64-bit-int array vs an out-of-range threshold stays
// approximate (a pathological case). array-vs-array never has a const, so it stays integer.
depth = base; wide = CV_64F; break;
case OP_MUL: case OP_DIV:
{
depth = base;
const bool w = (depth==CV_32U || depth==CV_32S || depth==CV_64U ||
depth==CV_64S || depth==CV_64F);
wide = w ? CV_64F : CV_32F;
if (op == OP_DIV)
{
// guard b==0 -> 0 when BOTH operands are integer (matches cv::divide); a flexible const
// counts as integer iff its value is integral. promote2 can float two wide ints, so this
// must read the ORIGINAL operands, not `depth`.
auto intOperand = [&](int s, int nd) {
if (nd != EW_DEPTH_NONE) return !isFloatDepth(nd);
AutoBuffer<double, MAX_LOCAL_CN> v; constDoubles(*this, s, v);
return v[0] == std::floor(v[0]); };
divGuard = intOperand(a, nd0) && intOperand(b, nd1);
}
break;
}
case OP_POW: case OP_HYPOT: case OP_ATAN2:
depth = (base == CV_64F) ? CV_64F : CV_32F; wide = depth; break;
default: // OP_ADD, OP_SUB
depth = base; wide = safeWide(depth); break;
}
// a flexible const that can't be represented exactly at `depth` (e.g. 2.5 in a u8 op) forces the
// wide working path; a NON-INTEGRAL const additionally forces a FLOAT compute (an integer wide
// would quantize the fraction - e.g. add(u8, 1.7) must not round 1.7 to 2).
auto fracConst = [&](int s) {
if (!isFlexConst(*this, s)) return false;
AutoBuffer<double, MAX_LOCAL_CN> v; int n = constDoubles(*this, s, v);
for (int ch = 0; ch < n; ch++)
if (v[ch] != std::floor(v[ch])) return true;
return false;
};
// BITWISE never widens or floats: a bit-pattern op keeps the array's own integer type, and an
// out-of-range/fractional scalar is just saturate/round-cast into it (matching scalarToRawData /
// cv::bitwise's classic convertAndUnrollScalar). Skipping the widening below leaves depth == result.
if (cat != CAT_BITWISE && (!constFits(*this, a, depth) || !constFits(*this, b, depth)))
{
depth = wide;
const bool frac = fracConst(a) || fracConst(b);
// MIN/MAX just SELECT an operand, so a fractional scalar threshold is round-cast into the
// array type (min(u8, 3.7) == min(u8, 4)) - matching the classic cv::min/max - instead of
// promoting the whole op to float (add/sub/absdiff DO need the float path to keep the fraction).
if (frac && !isFloatDepth(depth) && op != OP_MIN && op != OP_MAX)
depth = wide = (result == CV_64F || base == CV_64F) ? CV_64F : CV_32F;
// mul/div carry a scale-like float const at full f64 precision (cv::multiply/divide compute in
// f64), so don't settle for f32 - e.g. 110 * 147.2863... must round to 16201, not 16202.
if (frac && (op == OP_MUL || op == OP_DIV) && depth == CV_32F)
depth = wide = CV_64F;
}
int s0 = isFlexConst(*this, a) ? typedConstFrom(a, depth) : a;
int s1 = isFlexConst(*this, b) ? typedConstFrom(b, depth) : b;
int c0 = maybeAddCast(s0, depth), c1 = maybeAddCast(s1, depth);
// emit `depth -> result` directly when a kernel exists, else compute in `wide` and cast down.
// OP_DIV resolves via getDivFunc (carrying the /0 guard); every other op via getElemwiseFunc.
auto resolve = [&](int rd) {
return op == OP_DIV ? getDivFunc(depth, rd, divGuard)
: getElemwiseFunc(op, depth, depth, EW_DEPTH_NONE, rd);
};
TKernel k = resolve(result);
if (k.fptr)
{
int res = addTemp(result);
addInsn(op, c0, c1, 0, res, k, params);
return res;
}
k = resolve(wide);
CV_Assert(k.fptr && "ew: no kernel for this op/type combination");
int w = addTemp(wide);
addInsn(op, c0, c1, 0, w, k, params);
return maybeAddCast(w, result);
}
// ---------------------------------------------------------------------------
// TExpr::emitUnary(): type policy for a 1-input op. MATH ops (sqrt/exp/...) compute in the float
// domain; NEG/ABS keep the operand type; NOT stays integer. (OP_CAST is not routed here - an explicit
// cast is just maybeAddCast / a typed addConst at the call site.)
// ---------------------------------------------------------------------------
int TExpr::emitUnary(TOp op, int a, int rdepth, const Scalar& params)
{
const int nd = isFlexConst(*this, a) ? EW_DEPTH_NONE : arginfo[a].depth;
const ElemwiseCategory cat = opCategory(op);
// NEG and ABS have no kernels of their own - they are compositions over the binary family
// with a zero constant (flexible, so emitBinary types it as the operand's own type).
if (op == OP_NEG)
return emitBinary(OP_SUB, addConst(EW_DEPTH_NONE, Scalar(0.), 1), a, rdepth, params);
if (op == OP_ABS)
{
// peephole: abs(x - y) -> absdiff(x, y), ALWAYS. Strictly speaking the two differ on
// integers - the literal subtract saturates first (u8: max(x-y, 0); signed: clipped
// difference), absdiff computes the true |x - y| - but whoever writes abs(a - b) MEANS
// absdiff; the saturation artifacts are never the desired result. So we deliberately
// "don't notice" the difference and hand out the useful semantics. The sub is necessarily
// the last instruction and its result the last temp (abs is emitted right after its
// argument) - retire both, the moveToOutput manoeuvre.
if (!prog.empty() && arginfo[a].kind == TEMP &&
prog.back().op == OP_SUB && prog.back().result == a &&
arginfo[a].index == ntemps - 1)
{
const int x = prog.back().arg0, y = prog.back().arg1;
prog.pop_back();
arginfo[a].kind = NONE;
ntemps--;
return emitBinary(OP_ABSDIFF, x, y, rdepth, params);
}
// abs IS absdiff(a, 0), including the auto result type: a signed |a| lands in the
// UNSIGNED type of the same width (|-128| = 128 fits u8 exactly; pinning the result to
// the signed operand type would saturate it to 127). Fully uniform with the peephole.
return emitBinary(OP_ABSDIFF, a, addConst(EW_DEPTH_NONE, Scalar(0.), 1), rdepth, params);
}
int result = rdepth;
if (result == EW_DEPTH_NONE)
{
switch (cat)
{
// math is T -> T over the float depths (f16/bf16/f32/f64 kernels exist natively);
// an integer input computes - and lands - in the float domain
case CAT_MATH: result = isFloatDepth(nd) ? nd : CV_32F; break;
case CAT_BITWISE: result = (nd == EW_DEPTH_NONE) ? CV_32S : nd; break; // NOT
default: result = (nd == EW_DEPTH_NONE) ? CV_32F : nd; break; // NEG, ABS
}
}
int depth, wide;
switch (cat)
{
case CAT_MATH:
// T == result over a float depth => the native kernel (incl. the f16/bf16 in-kernel f32
// hub - no materialized f32 temps). Everything else computes in f32/f64 and casts.
depth = (isFloatDepth(nd) && nd == result) ? nd
: (nd == CV_64F || result == CV_64F) ? CV_64F : CV_32F;
wide = depth;
break;
case CAT_BITWISE: depth = result; wide = result; break;
default: depth = (nd == EW_DEPTH_NONE) ? result : nd; wide = result; break; // NEG/ABS
}
if (!constFits(*this, a, depth)) depth = (cat == CAT_MATH) ? depth : CV_32F;
int s0 = isFlexConst(*this, a) ? typedConstFrom(a, depth) : a;
int c0 = maybeAddCast(s0, depth);
TKernel k = getElemwiseFunc(op, depth, EW_DEPTH_NONE, EW_DEPTH_NONE, result);
if (k.fptr)
{
int res = addTemp(result);
addInsn(op, c0, 0, 0, res, k, params);
return res;
}
k = getElemwiseFunc(op, depth, EW_DEPTH_NONE, EW_DEPTH_NONE, wide);
CV_Assert(k.fptr && "ew: no kernel for this op/type combination");
int w = addTemp(wide);
addInsn(op, c0, 0, 0, w, k, params);
return maybeAddCast(w, result);
}
// ---------------------------------------------------------------------------
// TExpr::emitTernary(): clamp(x,lo,hi) and select(mask,x,y). clamp brings all three data operands to
// a common type and emits there; select keeps arg0 (the mask) untouched and only unifies the two
// branches. (Both are minimal: not yet test-covered, no kernels wired up.)
// ---------------------------------------------------------------------------
int TExpr::emitTernary(TOp op, int a, int b, int c, int rdepth)
{
auto dep = [&](int s) { return isFlexConst(*this, s) ? EW_DEPTH_NONE : arginfo[s].depth; };
if (op == OP_SELECT) // select(mask=a, x=b, y=c)
{
// the kernel consumes a 1-byte mask as-is (u8/s8/bool, mask != 0 semantics). Any other
// mask type - a wider array or a literal - is normalized by an explicit `mask != 0`
// compare (u8 result), NOT by a value cast (which would saturate/round the values).
if (isFlexConst(*this, a) || CV_ELEM_SIZE1(arginfo[a].depth) != 1)
a = emitBinary(OP_CMP_NE, a, addConst(EW_DEPTH_NONE, Scalar(0.), 1), CV_8U, Scalar());
int result = rdepth != EW_DEPTH_NONE ? rdepth : promoteArith(dep(b), dep(c));
if (result == EW_DEPTH_NONE) result = CV_32F;
// a literal branch (select(m, x, 0)) is a flexible const: type it at `result` directly
// (a value conversion in constbuf), never through an OP_CAST of a depth-less slot
int sb = isFlexConst(*this, b) ? typedConstFrom(b, result) : maybeAddCast(b, result);
int sc = isFlexConst(*this, c) ? typedConstFrom(c, result) : maybeAddCast(c, result);
int res = addTemp(result);
addInsn(OP_SELECT, a, sb, sc, res);
return res;
}
// clamp(x,lo,hi): unify all three operands at `result` (compute == result, no wide fallback).
// The x operand dominates the auto type: clamp(u8_img, 10, 200) must stay u8, and literal
// bounds are flexible consts typed at `result` directly (never OP_CAST of a depth-less slot).
int result = rdepth != EW_DEPTH_NONE ? rdepth
: promoteArith(promoteArith(dep(a), dep(b)), dep(c));
if (result == EW_DEPTH_NONE) result = CV_32F;
auto typed = [&](int s) {
return isFlexConst(*this, s) ? typedConstFrom(s, result) : maybeAddCast(s, result);
};
int c0 = typed(a), c1 = typed(b), c2 = typed(c);
TKernel k = getElemwiseFunc(op, result, result, result, result);
CV_Assert(k.fptr && "ew: no kernel for this op/type combination");
int res = addTemp(result);
addInsn(op, c0, c1, c2, res, k);
return res;
}
// ---------------------------------------------------------------------------
// TExpr::moveToOutput(): land `temp` in the existing slot `out`. Prefer redirecting `temp`'s single
// producer to write `out` directly (no copy), dropping `temp` when it was the last-added slot so
// ntemps stays minimal (keeps compile()'s no-temp fast exit for single-op programs). Otherwise copy.
// ---------------------------------------------------------------------------
int TExpr::moveToOutput(int temp, int out)
{
const int ninsn = (int)prog.size();
int producer = -1;
bool usedAsArg = false;
for (int i = 0; i < ninsn; i++)
{
const TExpr::Insn& ins = prog[i];
if (ins.result == temp) producer = i;
if (ins.arg0 == temp || ins.arg1 == temp || ins.arg2 == temp) usedAsArg = true;
}
if (arginfo[temp].kind == TEMP && producer >= 0 && !usedAsArg &&
arginfo[temp].depth == arginfo[out].depth)
{
// MOVE semantics: redirect `temp`'s single producer to write `out` directly, then leave the
// source slot EMPTY - reclassify it to NONE. A NONE slot gets no physical buffer (compile's
// liveness only walks TEMP slots) and is skipped everywhere in exec, so no stale dead-TEMP slot
// remains (which, buffer or not, would still cost per-tile setup in the general path) - and no
// slot removal / index renumbering is needed. This keeps single-op programs at zero temps =>
// compile()'s no-temp early-out (zero heap traffic) still fires. Every caller moves the result
// it JUST emitted, so `temp` is always the last-created temp - decrement ntemps to retire its
// index and keep the temp indices dense (0..ntemps-1) for compile()'s liveness arrays.
CV_Assert(arginfo[temp].index == ntemps - 1);
prog[producer].result = out;
arginfo[temp].kind = NONE;
ntemps--;
return out;
}
addInsn(OP_CAST, temp, 0, 0, out); // same/different depth copy into `out`
return out;
}
// ---------------------------------------------------------------------------
// TExpr::output(): declare a fresh OUTPUT of `rootSlot`'s depth and moveToOutput into it.
// ---------------------------------------------------------------------------
int TExpr::output(int rootSlot)
{
return moveToOutput(rootSlot, addOutput(arginfo[rootSlot].depth));
}
// ---------------------------------------------------------------------------
// TExpr::compile(): finalize the already-typed program. Bind every instruction's kernel (skipping
// any whose pointer was pre-set, e.g. div's caller-known /0 policy) and pack the temps into a
// minimal set of reusable physical buffers (liveness). All per-arg bookkeeping here is transient.
// ---------------------------------------------------------------------------
void TExpr::compile()
{
const int ninsn = (int)prog.size();
// Kernels are bound eagerly by addInsn at build time, so there is no binding pass here - compile()
// only counts consts and packs temp buffers.
// Materialize consts: convert each live CONST's source values (still in `srcdepth` at constofs)
// to its resolved `depth`, appending the result to constbuf; constofs then points at the converted
// values and srcdepth becomes the resolved depth. Flexible literals (depth==NONE) are dead - the
// emit* layers replaced them with typed copies - and are skipped. nconsts sizes exec's per-const
// header buffer and gates the fast path. (getConvertFunc may reallocate constbuf, so snapshot the
// source bytes first.)
const int nslots = (int)arginfo.size();
nconsts = 0;
for (int s = 1; s < nslots; s++)
{
Arg& a = arginfo[s];
if (a.kind != CONST || a.depth == EW_DEPTH_NONE) continue;
const int cn = std::max(1, a.channels), sd = a.srcdepth, dd = a.depth;
const size_t sesz = CV_ELEM_SIZE1(sd), desz = CV_ELEM_SIZE1(dd);
AutoBuffer<uchar, 64> srcbytes((size_t)cn * sesz);
memcpy(srcbytes.data(), (const uchar*)(constbuf.data() + a.constofs), (size_t)cn * sesz);
const size_t ofs = appendConstBuf(constbuf, dd, nullptr, cn); // reserve converted region
uchar* dst = (uchar*)(constbuf.data() + ofs);
if (sd == dd) memcpy(dst, srcbytes.data(), (size_t)cn * desz);
else getConvertFunc(sd, dd)(srcbytes.data(), 0, nullptr, 0, dst, 0, Size(cn, 1), nullptr);
a.constofs = ofs; a.srcdepth = dd;
nconsts++;
}
// ---- liveness: pack temps into a minimal set of reusable physical buffers. A program with no
// temps (single-op add/sub/mul/...) needs none of this - early out with ZERO heap traffic,
// so building such a program (the dominant cv::add-style call) allocates nothing. The temp
// case uses stack-backed AutoBuffers (inline for typical small expressions). ----
nbuffers = 0;
bufEszPrefix.resize(1); bufEszPrefix[0] = 0;
capElems = INT_MAX; // no temps => exec runs each tile in one block, no scratch
if (ntemps == 0) return;
AutoBuffer<int, 32> tempOfSlot(nslots);
for (int s = 0; s < nslots; s++) tempOfSlot[s] = -1;
for (int s = 1; s < nslots; s++)
if (arginfo[s].kind == TEMP) tempOfSlot[s] = arginfo[s].index;
AutoBuffer<int, 16> lastUse(ntemps);
for (int t = 0; t < ntemps; t++) lastUse[t] = -1;
for (int i = 0; i < ninsn; i++)
{
const TExpr::Insn& ins = prog[i];
int as[3] = { ins.arg0, ins.arg1, ins.arg2 };
for (int k = 0; k < 3; k++)
if (tempOfSlot[as[k]] >= 0) lastUse[tempOfSlot[as[k]]] = i;
}
bufferOfTemp.resize(ntemps);
for (int t = 0; t < ntemps; t++) bufferOfTemp[t] = -1;
AutoBuffer<int, 16> freeBufs(ntemps);
int nfree = 0, nbuf = 0;
for (int i = 0; i < ninsn; i++)
{
const TExpr::Insn& ins = prog[i];
int rt = tempOfSlot[ins.result];
if (rt >= 0 && bufferOfTemp[rt] < 0)
bufferOfTemp[rt] = nfree > 0 ? freeBufs[--nfree] : nbuf++;
int as[3] = { ins.arg0, ins.arg1, ins.arg2 };
for (int k = 0; k < 3; k++)
{
int t = tempOfSlot[as[k]];
if (t >= 0 && lastUse[t] == i)
{
freeBufs[nfree++] = bufferOfTemp[t];
lastUse[t] = -1; // the same temp may be several args of this insn ("d*d") -
// release its buffer once, or freeBufs overflows/double-frees
}
}
}
nbuffers = nbuf;
// Byte layout of the physical temp buffers, per output element: prefix sums of each buffer's
// max element size. bufEszPrefix[b]*region = buffer b's byte offset in the scratch; [nbuffers] =
// total temp bytes/element. Precomputed here (depths are build-time) so exec()/its fast path
// never recompute it per call, and the fast path can size its scratch in O(1).
bufEszPrefix.resize(nbuffers + 1);
for (int b = 0; b <= nbuffers; b++) bufEszPrefix[b] = 0;
for (int s = 1; s < nslots; s++)
if (arginfo[s].kind == TEMP)
{
int b = bufferOfTemp[arginfo[s].index], e = (int)CV_ELEM_SIZE1(arginfo[s].depth);
if (e > bufEszPrefix[b + 1]) bufEszPrefix[b + 1] = e; // max elem size in buffer b
}
for (int b = 0; b < nbuffers; b++) bufEszPrefix[b + 1] += bufEszPrefix[b]; // -> prefix sums
// L1 fragment cap: # elements one ~16KB scratch fragment holds (exec fragments the strip so the
// intermediates stay hot). Pure function of the temp byte layout, hence computed here once.
const int totalEsz = bufEszPrefix[nbuffers];
capElems = totalEsz > 0 ? std::max(64, (16 * 1024) / totalEsz) : INT_MAX;
}
// ============================ executor + builders (was ew_exec.cpp) ============================
// Logical shape of a Mat with channels as the innermost dimension; steps in elemsize1 units.
// Used here only to infer the broadcast RESULT shape (spatial + channels) for output allocation;
// broadcastOp does its own channel-aware layout for the traversal itself.
static void matLogical(const Mat& m, MatShape& shp, EwSteps& step, int& esz1)
{
esz1 = (int)m.elemSize1();
int nd = m.dims, cn = m.channels();
shp.resize(nd + 1);
for (int i = 0; i < nd; i++)
{
shp[i] = m.size[i];
step[i] = m.step[i] / esz1;
}
shp[nd] = cn;
step[nd] = 1;
}
// numpy-style broadcast of several right-aligned shapes.
static bool broadcastShape(const MatShape* shps, int K, MatShape& out)
{
int nd = 0;
for (int k = 0; k < K; k++) nd = std::max(nd, shps[k].dims);
out.assign(nd, 1);
for (int k = 0; k < K; k++)
{
const MatShape& s = shps[k];
int off = nd - s.dims;
for (int i = 0; i < s.dims; i++)
{
int d = s[i], &o = out[off + i];
if (o == 1) o = d;
else if (d != 1 && d != o) return false;
}
}
return true;
}
// Rough per-element cost of an op for the parallel_for_ stripe-count hint, in units of ~1/4 cycle
// per element of the VECTORIZED kernel (measured on the SIMD paths; e.g. the atan2/exp polynomials
// run at ~1.5 cycles/element, not the ~30 a scalar-cost model would suggest - the old values
// over-split the work into stripes too small to amortize the per-job dispatch overhead).
// Unknown ops default to ~division. cv::expression will own this once it drives broadcastOp.
static int opCost(TOp op)
{
switch (op)
{
case OP_ADD: case OP_SUB: case OP_MUL: case OP_MIN: case OP_MAX:
case OP_ABSDIFF: case OP_AND: case OP_OR: case OP_XOR: case OP_NOT:
case OP_NEG: case OP_ABS: case OP_CAST: case OP_RELU: case OP_SELECT:
case OP_CMP_EQ: case OP_CMP_NE: case OP_CMP_LT:
case OP_CMP_LE: case OP_CMP_GT: case OP_CMP_GE: return 1;
case OP_DIV: case OP_SQRT: case OP_HYPOT: case OP_CONVERT_SCALE: return 3;
case OP_SIN: case OP_COS: case OP_TANH: case OP_ERF: case OP_ATAN2:
case OP_EXP: case OP_LOG: case OP_POW: return 6;
default: return 3;
}
}
TExpr::TExpr()
{
clear();
}
// Reset to an empty program: drop all instructions/slots and re-seat slot 0 = NONE.
void TExpr::clear()
{
prog.allocate(0);
arginfo.allocate(0);
bufferOfTemp.allocate(0);
constbuf.allocate(0);
ninputs = noutputs = ntemps = nbuffers = 0;
Arg none; // slot 0: the reserved empty operand (kind == NONE)
arginfo.push_back(none);
}
// Human-readable dump of the program (slot table + instruction list). Const values are shown in f64.
void TExpr::dump(std::ostream& os) const
{
auto dn = [](int d) -> const char* {
switch (d) {
case EW_DEPTH_NONE: return "flex";
case CV_8U: return "u8"; case CV_8S: return "s8";
case CV_16U: return "u16"; case CV_16S: return "s16";
case CV_32U: return "u32"; case CV_32S: return "s32";
case CV_64U: return "u64"; case CV_64S: return "s64";
case CV_16F: return "f16"; case CV_16BF:return "bf16";
case CV_32F: return "f32"; case CV_64F: return "f64";
case CV_Bool:return "bool";
default: return "?"; }
};
auto kn = [](ArgKind k) -> const char* {
switch (k) { case NONE: return "none"; case INPUT: return "in"; case CONST: return "const";
case TEMP: return "temp"; case OUTPUT: return "out"; } return "?";
};
os << "TExpr: inputs=" << ninputs << " outputs=" << noutputs << " temps=" << ntemps
<< " buffers=" << nbuffers << " consts=" << nconsts << " insns=" << (int)prog.size() << "\n";
os << " slots:\n";
for (int s = 0; s < (int)arginfo.size(); s++) {
const Arg& a = arginfo[s];
os << " [" << s << "] " << kn(a.kind);
if (a.kind != NONE) os << " " << dn(a.depth);
if (a.kind == INPUT || a.kind == OUTPUT || a.kind == TEMP) os << " idx=" << a.index;
if (a.kind == CONST) {
const int cn = std::max(1, a.channels);
AutoBuffer<double, MAX_LOCAL_CN> v(cn);
const uchar* src = (const uchar*)(constbuf.data() + a.constofs);
if (a.srcdepth == CV_64F) memcpy(v.data(), src, (size_t)cn * sizeof(double));
else getConvertFunc(a.srcdepth, CV_64F)(src, 0, nullptr, 0, (uchar*)v.data(), 0, Size(cn, 1), nullptr);
os << " cn=" << cn << " src=" << dn(a.srcdepth) << " {";
for (int c = 0; c < cn; c++) os << (c ? "," : "") << v[c];
os << "}";
}
os << "\n";
}
os << " prog:\n";
for (int i = 0; i < (int)prog.size(); i++) {
const Insn& ins = prog[i];
os << " " << i << ": " << opName(ins.op) << "(" << ins.arg0;
if (ins.arg1) os << ", " << ins.arg1;
if (ins.arg2) os << ", " << ins.arg2;
os << ") -> " << ins.result;
if (ins.kernel.flags) os << " kflags=" << ins.kernel.flags;
if (ins.params[0] != 1.0 || ins.params[1] != 0.0)
os << " params=[" << ins.params[0] << "," << ins.params[1] << "," << ins.params[2] << "]";
if (!ins.kernel.fptr) os << " [UNBOUND]";
os << "\n";
}
}
// ---- program builders: append a slot / instruction, return its index ----
int TExpr::addInput(int depth)
{
Arg a; a.kind = INPUT; a.depth = depth; a.index = ninputs++;
arginfo.push_back(a); return (int)arginfo.size() - 1;
}
int TExpr::addOutput(int depth)
{
Arg a; a.kind = OUTPUT; a.depth = depth; a.index = noutputs++;
arginfo.push_back(a); return (int)arginfo.size() - 1;
}
int TExpr::addTemp(int depth)
{
Arg a; a.kind = TEMP; a.depth = depth; a.index = ntemps++;
arginfo.push_back(a); return (int)arginfo.size() - 1;
}
// Source = a cv::Scalar (f64, up to 4 channels): stored as CV_64F in constbuf.
int TExpr::addConst(int depth, const Scalar& v, int channels)
{
CV_Assert(channels <= 4);
Arg a; a.kind = CONST; a.depth = depth; a.channels = channels; a.srcdepth = CV_64F;
a.constofs = appendConstBuf(constbuf, CV_64F, v.val, channels);
arginfo.push_back(a); return (int)arginfo.size() - 1;
}
// Source = native bytes of any depth/channel-count (e.g. a Vec<_,N> scalar): stored as-is.
int TExpr::addConst(int depth, int srcdepth, const void* data, int channels)
{
// A CONST is a broadcast scalar - capped at 4 channels (a Scalar). Need more? Pass a 0-D Mat with
// the desired channel count as an INPUT: broadcasting handles it, and it isn't limited to 4.
CV_Assert(channels <= 4);
Arg a; a.kind = CONST; a.depth = depth; a.channels = channels; a.srcdepth = srcdepth;
a.constofs = appendConstBuf(constbuf, srcdepth, data, channels);
arginfo.push_back(a); return (int)arginfo.size() - 1;
}
// A typed copy of flexible CONST `srcSlot` at the resolved `depth`, sharing its (still-source)
// values in constbuf; compile() converts each such slot's values to its `depth`.
int TExpr::typedConstFrom(int srcSlot, int depth)
{
const Arg& src = arginfo[srcSlot];
Arg a; a.kind = CONST; a.depth = depth; a.channels = src.channels;
a.srcdepth = src.srcdepth; a.constofs = src.constofs; // shares the source region
arginfo.push_back(a); return (int)arginfo.size() - 1;
}
// Append one instruction with a pre-resolved kernel (the caller probed getElemwiseFunc, or knows
// the kernel - e.g. div's /0-aware kernel). No re-resolution.
int TExpr::addInsn(TOp op, int a0, int a1, int a2, int result, const TKernel& kernel, const Scalar& params)
{
TExpr::Insn ins; ins.op = op; ins.arg0 = a0; ins.arg1 = a1; ins.arg2 = a2; ins.result = result;
ins.params = params; ins.kernel = kernel;
prog.push_back(ins);
return (int)prog.size() - 1;
}
// Append one instruction, resolving its kernel NOW from the operand/result depths (final at build
// time). compile() therefore never re-resolves. A builder that needs a specific kernel (div's /0
// policy) pushes a pre-bound Insn directly instead.
int TExpr::addInsn(TOp op, int a0, int a1, int a2, int result, const Scalar& params)
{
int d0 = a0 ? arginfo[a0].depth : EW_DEPTH_NONE;
int d1 = a1 ? arginfo[a1].depth : EW_DEPTH_NONE;
int d2 = a2 ? arginfo[a2].depth : EW_DEPTH_NONE;
TKernel k = getElemwiseFunc(op, d0, d1, d2, arginfo[result].depth);
CV_Assert(k.fptr && "ew: no kernel for this op/type combination");
return addInsn(op, a0, a1, a2, result, k, params);
}
// Cast `arg` to `depth` only if it is not already that depth (the sole cast-insertion helper for
// the hand builders and the emit* layers); returns the slot holding the value at `depth`.
int TExpr::maybeAddCast(int arg, int depth)
{
if (arginfo[arg].depth == depth) return arg;
int t = addTemp(depth);
addInsn(OP_CAST, arg, 0, 0, t);
return t;
}
// Run one resolved instruction over a width x height tile. Every kernel uses ONE calling
// convention - the universal KernelFunc (element steps, the instruction's params block, plus
// kernel.flags/kernel.userdata). OP_CAST / OP_CONVERT_SCALE bind castKernel, which forwards to a
// core convert BinaryFunc (carried in kernel.userdata) over the distinct sub-region and then
// expands it across any broadcast axis (see castKernel/expandKernel in ew_kernels.cpp).
static inline void runInsn(const TExpr::Insn& ins,
const void* p0, size_t y0, size_t x0,
const void* p1, size_t y1, size_t x1,
const void* p2, size_t y2, size_t x2,
void* pr, size_t yr, int w, int h)
{
int code = ins.kernel.fptr(p0, y0, x0, p1, y1, x1, p2, y2, x2, pr, yr, w, h,
ins.params.val, ins.kernel.flags, ins.kernel.userdata);
CV_Assert(code >= 0);
}
struct EwBody {
const TExpr::Insn* prog;
// slot -> a NON-NEGATIVE index whose meaning is given by slotKind[s]:
// TExpr::INPUT / TExpr::OUTPUT : index into tile.slices (the broadcast operand list; consts,
// now 0-dim broadcast operands, are relabeled INPUT and live here)
// TExpr::TEMP : physical temp-buffer id
const int* slotMap;
const signed char* slotKind; // [nslots] arginfo[s].kind, for resolving slotMap[s]
const int* bufEszPrefix; // [nbuffers+1] prefix sums of temp elem sizes; [nbuffers] = total
int ninsn, nslots, nbuffers, capElems; // capElems: L1 fragment element cap (0 if no temps)
};
void TExpr::outputShape(const Mat* const* inputs, MatShape& spatial, int& channels) const
{
// Fast path: every real input shares one shape + channel count => the result IS inputs[0]'s
// (consts add no dims/channels). Otherwise re-broadcast the logical (spatial + channel) shapes.
const int rcn0 = ninputs >= 1 ? inputs[0]->channels() : 1;
bool sameShape = ninputs >= 1;
for (int i = 1; sameShape && i < ninputs; i++)
{
const Mat& a = *inputs[i];
if (a.dims != inputs[0]->dims || a.channels() != rcn0) sameShape = false;
else for (int d = 0; d < a.dims; d++) if (a.size[d] != inputs[0]->size[d]) { sameShape = false; break; }
}
if (sameShape) { spatial = inputs[0]->size; channels = rcn0; return; }
AutoBuffer<MatShape, 16> bshapes(std::max(ninputs, 1));
MatShape shp; EwSteps step; int esz1;
for (int i = 0; i < ninputs; i++) { matLogical(*inputs[i], shp, step, esz1); bshapes[i] = shp; }
MatShape full;
CV_Assert(broadcastShape(bshapes.data(), ninputs, full) && "ew: inputs not broadcast-compatible");
const int ndFull = (int)full.size();
channels = full[ndFull - 1];
spatial = full; spatial.resize(ndFull - 1);
}
// Convenience overload: inputs as a contiguous Mat array -> build the pointer array + forward.
void TExpr::outputShape(const Mat* inputs, MatShape& spatial, int& channels) const
{
AutoBuffer<const Mat*, 16> ptrs(std::max(ninputs, 1));
for (int i = 0; i < ninputs; i++) ptrs[i] = &inputs[i];
outputShape(ptrs.data(), spatial, channels);
}
// At namespace scope, NOT inside exec(): MSVC 2019 loses the constexpr-ness of function-local
// constants used as template arguments inside a lambda (C2975).
static constexpr int LOCAL_HDRS = 3;
static constexpr int LOCAL_CONSTS = 4;
static constexpr int LOCAL_OPS = 16;
void TExpr::exec(const Mat* const* inputs, Mat* outputs)
{
using BrTile = BroadcastOp::Tile;
using BrSlice = BroadcastOp::Slice;
const int nslots = (int)arginfo.size();
CV_Assert(nslots >= 1 && arginfo[0].kind == TExpr::NONE);
// ---- cheap fast path: a const-free program over small, same-shape, continuous arrays. Channels
// fold into one flat contiguous run, so the whole job is a flat (total x 1) strip - run the
// program directly here and skip the per-call prep loop, broadcastOp's geometry, 2D tiling
// and the parallel framework (all pure overhead at this size). Temps are allowed: their
// byte layout (bufEszPrefix) is known from compile(), so we size an L1 scratch in O(1) and
// walk the strip in L1-sized fragments (the intermediates stay hot). nconsts/nbuffers/
// bufEszPrefix come from compile(); the rest is a quick check of the args. ----
if (nconsts == 0 && ninputs >= 1)
{
constexpr size_t EW_FASTPATH_MAX = 1u << 17; // above this the tiled/parallel path wins
const Mat& r = *inputs[0];
const int rcn = r.channels();
bool ok = r.isContinuous();
for (int i = 1; ok && i < ninputs; i++)
{
const Mat& a = *inputs[i];
ok = ok && a.isContinuous() && a.channels() == rcn && a.size == r.size;
}
// the flat strip writes the output(s) contiguously too, so it can't serve a non-contiguous
// (cropped-ROI) destination - those fall to the general strided path below (an empty output,
// which exec will allocate contiguous, counts as continuous here).
for (int s = 1; ok && s < nslots; s++)
if (arginfo[s].kind == OUTPUT) ok = ok && outputs[arginfo[s].index].isContinuous();
const size_t total = ok ? r.total() * rcn : 0;
if (ok && total <= EW_FASTPATH_MAX)
{
const MatShape rshape = r.size; // spatial dims (channels separate), as a MatShape
for (int s = 1; s < nslots; s++)
if (arginfo[s].kind == OUTPUT)
outputs[arginfo[s].index].create(rshape, CV_MAKETYPE(arginfo[s].depth, rcn));
const int ninsn = (int)prog.size();
if (nbuffers == 0)
{
// No temps (single-op add/sub/mul/...): one flat (total x 1) pass, operands point
// straight at the Mat data - no scratch, no fragment loop. Referenced slots are INPUT/
// OUTPUT (a moved-from NONE slot may exist but is never an instruction argument).
auto ptrOf = [&](int s) -> void* {
if (s <= 0) return nullptr;
const Arg& ai = arginfo[s];
if (ai.kind == INPUT) return (void*)inputs[ai.index]->data;
if (ai.kind == OUTPUT) return (void*)outputs[ai.index].data;
return nullptr; // NONE
};
const int w = (int)total;
for (int n = 0; n < ninsn; n++)
{
const Insn& ins = prog[n];
runInsn(ins, ptrOf(ins.arg0), 0, 1, ptrOf(ins.arg1), 0, 1,
ptrOf(ins.arg2), 0, 1, ptrOf(ins.result), 0, w, 1);
}
return;
}
// Temps present: walk the strip in L1-sized fragments so the intermediates stay hot.
// Byte layout (bufEszPrefix) and the fragment cap (capElems) both come from compile();
// scratch is totalEsz*wf0 (<= ~16KB).
const int totalEsz = bufEszPrefix[nbuffers];
const int wf0 = std::min((int)total, capElems);
// Scratch for the temp buffers. AutoBuffer no longer value-inits its tail, so a fresh per-call
// buffer is free (we only WRITE to it); the inline 16KB covers the L1-capped size, heap backs
// the rare larger case.
AutoBuffer<uchar, 16*1024 + 256> scratchBuf((size_t)totalEsz * (size_t)wf0);
uchar* scratch = scratchBuf.data();
for (int x0 = 0; x0 < (int)total; x0 += wf0)
{
const int wf = std::min(wf0, (int)total - x0);
auto ptrOf = [&](int s) -> void* {
if (s <= 0) return nullptr;
const Arg& ai = arginfo[s];
size_t esz = CV_ELEM_SIZE1(ai.depth);
if (ai.kind == INPUT) return (uchar*)inputs[ai.index]->data + (size_t)x0 * esz;
if (ai.kind == OUTPUT) return (uchar*)outputs[ai.index].data + (size_t)x0 * esz;
if (ai.kind != TEMP) return nullptr; // NONE: moved-from
const int b = bufferOfTemp.empty() ? ai.index : bufferOfTemp[ai.index]; // TEMP
return scratch + (size_t)bufEszPrefix[b] * (size_t)wf0; // fragment-local
};
for (int n = 0; n < ninsn; n++)
{
const Insn& ins = prog[n];
runInsn(ins, ptrOf(ins.arg0), 0, 1, ptrOf(ins.arg1), 0, 1,
ptrOf(ins.arg2), 0, 1, ptrOf(ins.result), 0, wf, 1);
}
}
return;
}
}
// Scalars (TExpr::CONST) never influence the result shape or channel count - the output geometry
// comes from the real array inputs alone (a scalar broadcasts into whatever they produce).
// A flexible CONST (depth == EW_DEPTH_NONE) is a leftover literal: the emit* layers materialize a
// typed copy at each use, so the original is dead - skipped entirely (no header built for it).
// ---- 1. result shape (spatial dims + channel count), channels innermost ----
// Fast path: every input shares one shape+channels => the result IS inputs[0]'s shape. Skips
// the matLogical + broadcastShape rebuild/re-broadcast done only to size the output. Consts are
// irrelevant here (they don't add dims/channels), so they never break the fast path.
MatShape spatial;
int rchannels;
outputShape(inputs, spatial, rchannels);
// ---- 2. ONE pass over slots: allocate outputs, build the operand pointer list + slot map,
// per-slot element size, and per-temp-buffer element size. No Mat copies (arr[] holds
// pointers); only real inputs + outputs become broadcast operands. TExpr::CONST scalars are
// NOT broadcast operands and get NO Mat header: each is materialized once into a typed
// scratch (constStore) and exposed to the body as a fixed slice {ptr=&value, 0, 0} - a
// scalar the kernels read by broadcast. slotMap[s] stays a plain non-negative index;
// its meaning (arr / temp / const) is recovered from arginfo[s].kind (see EwBody).
// In-place: an input shares its data buffer with an output. An output create() may realloc that
// buffer while an input still needs the old contents, so we incref (header-copy) every input and
// read through the copies. Cheap data-pointer aliasing test (nullptr data => never matches).
bool inplace = false;
for (int i = 0; i < ninputs && !inplace; i++)
for (int j = 0; j < noutputs; j++)
if (inputs[i]->data && inputs[i]->data == outputs[j].data) { inplace = true; break; }
int nhdrs = inplace ? ninputs : 0;
AutoBuffer<const Mat*, LOCAL_OPS> arr(ninputs + noutputs + nconsts);
AutoBuffer<Mat, LOCAL_HDRS> hdrs(std::max(nhdrs, 1)); // non-owning input headers (in-place incref)
AutoBuffer<const Mat*, LOCAL_HDRS> inptr(std::max(nhdrs, 1)); // repointed input list (in-place only)
AutoBuffer<Mat, LOCAL_CONSTS> constHdrBuf(std::max(nconsts, 1)); // 0-dim headers over constbuf
AutoBuffer<int, LOCAL_OPS> slotMap(nslots);
AutoBuffer<signed char, LOCAL_OPS> slotKind(nslots);
int narr = 0, nc = 0;
if (inplace) {
// save (incref) inputs in the case of in-place operation
// to protect them from premature deallocation
for (int j = 0; j < ninputs; j++) { hdrs[j] = *inputs[j]; inptr[j] = &hdrs[j]; }
inputs = inptr.data();
}
slotKind[0] = (signed char)TExpr::NONE;
for (int s = 1; s < nslots; s++)
{
const TExpr::Arg& ai = arginfo[s];
slotKind[s] = (signed char)ai.kind;
if (ai.kind == TExpr::INPUT) {
slotMap[s] = narr;
arr[narr++] = inputs[ai.index];
}
else if (ai.kind == TExpr::OUTPUT) {
outputs[ai.index].create(spatial, CV_MAKETYPE(ai.depth, rchannels));
slotMap[s] = narr; arr[narr++] = &outputs[ai.index];
}
else if (ai.kind == TExpr::CONST && ai.depth == EW_DEPTH_NONE) {
slotKind[s] = (signed char)TExpr::NONE; // dead flexible literal: never referenced
slotMap[s] = 0;
}
else if (ai.kind == TExpr::CONST) {
// a materialized const rides the broadcast machinery as a 0-dim, per-channel operand: a
// multichannel const forces CH_DIM (per-channel scalars); a single value broadcasts
// everywhere. compile() already converted its values (in constbuf) to ai.depth.
const int c = std::max(1, ai.channels);
constHdrBuf[nc] = Mat(MatShape::scalar(), CV_MAKETYPE(ai.depth, c),
(void*)(constbuf.data() + ai.constofs));
slotKind[s] = (signed char)TExpr::INPUT; // to the body it is just an array operand
slotMap[s] = narr; arr[narr++] = &constHdrBuf[nc];
nc++;
}
else if (ai.kind == TExpr::TEMP) {
slotMap[s] = bufferOfTemp.empty() ? ai.index : bufferOfTemp[ai.index]; // physical buffer id
}
else { // TExpr::NONE: a moved-from temp (or dead literal) - inert, no operand, no buffer. Never
// referenced by any instruction; do NOT touch bufferOfTemp (its retired index may be >= ntemps).
slotMap[s] = 0;
}
}
// Temp-buffer byte layout (prefix sums, bufEszPrefix) and the L1 fragment cap (capElems) were
// both precomputed by compile(); the body uses them directly (prefix[buf]*region = a buffer's
// byte offset). capElems == INT_MAX when there are no temps => the body runs each tile in a
// single block (no scratch).
// ---- 3. ONE pass over instructions: summed per-element cost (kernels were bound at compile()). ----
const int ninsn = (int)prog.size();
long long costPerElem = 0;
for (int n = 0; n < ninsn; n++)
costPerElem += opCost(prog[n].op);
// ---- 4. parallel work hint: total output scalars x summed per-element op cost / budget. ----
long long otot = (long long)rchannels;
for (int d = 0; d < (int)spatial.size(); d++) otot *= spatial[d];
// ~4 stripes per thread is plenty of granularity for element-wise work; more only multiplies
// the per-job dispatch overhead (notably on the macOS/GCD backend). The absolute ceiling is
// 32 pieces, EXCEPT on machines with many (heterogeneous) cores: there anything coarser than
// ~3 pieces per worker makes the slow (efficiency) cores equal-share bottlenecks, so the
// ceiling grows as 3*nthreads instead.
// getNumThreads() is clamped from below: some backends may report 0 (WINRT, plugins).
const double T = (double)std::max(getNumThreads(), 1);
const double nstripes = std::min(
(double)otot * (double)std::max<long long>(costPerElem, 1) * (1./ (double)(1 << 18)),
std::min(4.*T, std::max(32., 3.*T)));
EwBody body;
body.prog = prog.data();
body.ninsn = ninsn;
body.slotMap = slotMap.data();
// ---- 5. drive: broadcastOp does geometry + 2D tiling + parallelism; the body runs the
// frozen program on each tile (temps tile-local, re-pointed from the tile slices).
// expandChannels=true => the body always sees single-channel data.
//
// All state the body needs is packed into one POD (EwBody) so the lambda captures a
// SINGLE reference: the closure is then one pointer, fits std::function's small-buffer
// and never heap-allocates. Inside, the hot fields are copied into locals so the
// per-tile/per-insn loops read them from registers, not through the captured pointer. -
body.bufEszPrefix = bufEszPrefix.data();
body.slotKind = slotKind.data();
body.nslots = nslots;
body.nbuffers = nbuffers;
body.capElems = capElems;
broadcastOp(arr.data(), narr, [&](const BrTile& tile)
{
EwBody& bc = body;
const TExpr::Insn* insns = bc.prog; // hot fields -> locals (registers/stack)
const int* const smap = bc.slotMap;
const signed char* const skind = bc.slotKind;
const int nin = bc.ninsn, nsl = bc.nslots;
const int w = tile.width, h = tile.height;
// Run the program over L1-sized 2D blocks so the intermediates stay in cache. ONE rule covers
// every tile shape: keep the strip as WIDE as fits L1 (long inner loop => good SIMD + hits the
// kernels' width-specific branches), then add as many rows as still fit (bw*bh <= capElems).
// For the dominant 1D tile (w huge, h==1) this is a column strip; for a tall-thin tile
// (w==channels, h huge - e.g. a masked op) it instead splits the LONG axis (height), so the
// temp block stays contiguous and the kernels keep full width. With NO temps capElems==INT_MAX
// => bw=w, bh=h: one block over the whole tile, region temp store is empty (zero bytes), and
// each operand slot is re-pointed once straight at its tile slice. Each temp buffer occupies
// [bufEszPrefix[buf]*region, ...) bytes in tstore; region = bw*bh.
const int* const eszPrefix = bc.bufEszPrefix;
const int bw = std::min(w, bc.capElems);
const int bh = std::min(h, std::max(1, bc.capElems / std::max(1, bw)));
const size_t region = alignSize((size_t)bw * bh, 8);
AutoBuffer<uchar, 16*1024 + 256> tstoreBuf((size_t)eszPrefix[bc.nbuffers] * region); // inline (<= ~16KB)
uchar* tstore = tstoreBuf.data();
AutoBuffer<BrSlice, LOCAL_OPS> args(nsl);
for (int y0 = 0; y0 < h; y0 += bh)
{
const int hf = std::min(bh, h - y0);
for (int x0 = 0; x0 < w; x0 += bw)
{
const int wf = std::min(bw, w - x0);
for (int s = 1; s < nsl; s++)
{
BrSlice& a = args[s];
const int k = skind[s];
size_t esz = CV_ELEM_SIZE1(arginfo[s].depth);
if (k == TExpr::TEMP) // contiguous block-local buffer
{
a.ptr = tstore + (size_t)eszPrefix[smap[s]] * region;
a.stepy = (size_t)wf*esz; a.stepx = 1;
}
else // array operand (incl. 0-dim consts): this
// block of the broadcast slice
{
const BrSlice& sl = tile.slices[smap[s]];
a.ptr = (uchar*)sl.ptr +
((size_t)y0 * sl.stepy + (size_t)x0 * sl.stepx) * esz;
a.stepy = sl.stepy*esz; a.stepx = sl.stepx;
}
}
for (int n = 0; n < nin; n++)
{
const TExpr::Insn& ins = insns[n];
const BrSlice& a0 = args[ins.arg0]; const BrSlice& a1 = args[ins.arg1];
const BrSlice& a2 = args[ins.arg2]; const BrSlice& rr = args[ins.result];
runInsn(ins, a0.ptr, a0.stepy, a0.stepx, a1.ptr, a1.stepy, a1.stepx,
a2.ptr, a2.stepy, a2.stepx, (void*)rr.ptr, rr.stepy, wf, hf);
}
}
}
}, true, nstripes);
}
// Convenience overload: inputs as a contiguous Mat array -> build the pointer array + forward.
void TExpr::exec(const Mat* inputs, Mat* outputs)
{
AutoBuffer<const Mat*, 16> ptrs(std::max(ninputs, 1));
for (int i = 0; i < ninputs; i++) ptrs[i] = &inputs[i];
exec(ptrs.data(), outputs);
}
// ---------------------------------------------------------------------------
// Manual program builders (stand-ins for the future engine-backed cv::add etc.): they skip the
// source graph and emit the program directly through TExpr's addInput/addTemp/addOutput/addInsn.
// ---------------------------------------------------------------------------
// Compose a binary op (ADD/SUB/MUL/DIV/MIN/MAX/ABSDIFF/CMP_*) for any (depth0, depth1, rdepth): just
// the operand/output plumbing around emitBinary, which owns the whole type policy (compute type, wide
// fallback, div's /0 guard, mul/div `scale` in params[0]). emitBinary returns the result slot;
// moveToOutput lands it in the output with no dead temp (so a single-op program keeps zero temps).
//
// maskDepth != EW_DEPTH_NONE adds a write-mask (input #2): the arithmetic result lands in a temp and a
// final select(mask, r, dst) -> dst overwrites only the masked subset of the (pre-existing) output,
// leaving the rest UNCHANGED (matching cv::add/... with a mask); the output slot rides as both the
// select's arg2 and its result (the kernel is alias-safe). select is always the LAST instruction.
// The mask is a single-channel 1-byte array (bool/u8/s8) the size of the output spatial shape; it
// rides the normal broadcast machinery, so nothing special is needed in the executor.
void makeBinaryArithProgram(TExpr& p, TOp op, int depth0, int depth1, int rdepth,
int maskDepth, double scale)
{
p.clear();
if (rdepth < 0) // rdepth == -1 => auto (like cv::'s dtype=-1)
{
if (opCategory(op) == CAT_COMPARE)
rdepth = CV_8U; // compare -> u8 mask
else
{
rdepth = promoteArith(depth0, depth1);
if (op == OP_ABSDIFF) rdepth = absdiffResultDepth(rdepth); // signed |a-b| -> unsigned same width
}
}
const bool masked = maskDepth != EW_DEPTH_NONE;
int sIn0 = p.addInput(depth0);
int sIn1 = p.addInput(depth1);
int sMask = masked ? p.addInput(maskDepth) : 0;
int sOut = p.addOutput(rdepth);
int r = p.emitBinary(op, sIn0, sIn1, rdepth, Scalar(scale));
if (masked)
p.addInsn(OP_SELECT, sMask, r, sOut, sOut); // dst = mask ? r : dst
else
p.moveToOutput(r, sOut); // straight into the output, no dead temp
p.compile(); // pack temp buffers + count consts (kernels already bound by addInsn)
}
// addWeighted(a, alpha, b, beta, gamma) = a*alpha + b*beta + gamma. One fused kernel (two v_fma) via
// emitBinary(OP_ADDW); a final cast is appended only when the requested rdepth isn't a type the kernel
// emits directly. alpha/beta/gamma travel in the instruction's params block, not as operands.
void makeAddWeightedProgram(TExpr& p, int depth0, int depth1, int rdepth,
double alpha, double beta, double gamma)
{
p.clear();
int sA = p.addInput(depth0);
int sB = p.addInput(depth1);
int sOut = p.addOutput(rdepth);
int r = p.emitBinary(OP_ADDW, sA, sB, rdepth, Scalar(alpha, beta, gamma));
p.moveToOutput(r, sOut);
p.compile(); // pack temp buffers + count consts (kernels already bound by addInsn)
}
// ============================ parser (was ew_parser.cpp) ============================
namespace {
// --- tokens ------------------------------------------------------------------------------
enum TokType { T_NUM, T_INPUT, T_IDENT, T_OP, T_LPAREN, T_RPAREN, T_COMMA, T_SEMI, T_ASSIGN, T_END };
struct Token
{
TokType type = T_END;
double num = 0;
int input = 0;
std::string text; // identifier or operator spelling
};
// --- lexer -------------------------------------------------------------------------------
struct Lexer
{
std::string_view s;
size_t pos = 0;
explicit Lexer(std::string_view src) : s(src) {}
static bool isIdentStart(char c) { return std::isalpha((unsigned char)c) || c == '_'; }
static bool isIdentChar(char c) { return std::isalnum((unsigned char)c) || c == '_'; }
Token next()
{
while (pos < s.size() && std::isspace((unsigned char)s[pos])) pos++;
Token t;
if (pos >= s.size()) { t.type = T_END; return t; }
char c = s[pos];
if (c == '{') // input placeholder {N}
{
pos++;
size_t start = pos;
while (pos < s.size() && s[pos] != '}') pos++;
CV_Assert(pos < s.size() && "ew::expression: unterminated '{'");
t.type = T_INPUT;
t.input = std::atoi(std::string(s.substr(start, pos - start)).c_str());
pos++; // consume '}'
return t;
}
if (std::isdigit((unsigned char)c) || (c == '.' && pos + 1 < s.size() &&
std::isdigit((unsigned char)s[pos + 1])))
{
char* end = nullptr;
std::string num(s.substr(pos));
t.type = T_NUM;
t.num = std::strtod(num.c_str(), &end);
pos += (size_t)(end - num.c_str());
return t;
}
if (isIdentStart(c))
{
size_t start = pos;
while (pos < s.size() && isIdentChar(s[pos])) pos++;
t.type = T_IDENT;
t.text.assign(s.substr(start, pos - start));
return t;
}
switch (c)
{
case '(': pos++; t.type = T_LPAREN; return t;
case ')': pos++; t.type = T_RPAREN; return t;
case ',': pos++; t.type = T_COMMA; return t;
case ';': pos++; t.type = T_SEMI; return t;
}
// multi/!single-char operators
auto two = [&](const char* op) {
return pos + 1 < s.size() && s[pos] == op[0] && s[pos + 1] == op[1];
};
t.type = T_OP;
if (two("<=") || two(">=") || two("==") || two("!=") || two("**"))
{ t.text.assign(s.substr(pos, 2)); pos += 2; return t; }
if (c == '=') { pos++; t.type = T_ASSIGN; return t; }
CV_Assert(std::strchr("+-*/<>&|^!?:", c) && "ew::expression: unexpected character");
t.text.assign(1, c); pos++;
return t;
}
};
// --- operator / function tables ----------------------------------------------------------
static int binPrec(const std::string& op)
{
if (op == "**") return 8; // power binds tighter than '*'; RIGHT-associative
if (op == "*" || op == "/") return 7;
if (op == "+" || op == "-") return 6;
if (op == "<" || op == "<=" || op == ">" || op == ">=") return 5;
if (op == "==" || op == "!=") return 4;
if (op == "&") return 3;
if (op == "^") return 2;
if (op == "|") return 1;
return -1;
}
static TOp binOp(const std::string& op)
{
if (op == "**") return OP_POW;
if (op == "+") return OP_ADD;
if (op == "-") return OP_SUB;
if (op == "*") return OP_MUL;
if (op == "/") return OP_DIV;
if (op == "<") return OP_CMP_LT;
if (op == "<=") return OP_CMP_LE;
if (op == ">") return OP_CMP_GT;
if (op == ">=") return OP_CMP_GE;
if (op == "==") return OP_CMP_EQ;
if (op == "!=") return OP_CMP_NE;
if (op == "&") return OP_AND;
if (op == "|") return OP_OR;
if (op == "^") return OP_XOR;
CV_Error(Error::StsParseError, "ew::expression: bad binary operator");
}
// type-cast function name -> depth, or -1 if not a type name
static int typeDepth(const std::string& name)
{
if (name == "float") return CV_32F;
if (name == "double") return CV_64F;
if (name == "half" || name == "float16") return CV_16F;
if (name == "bfloat16") return CV_16BF;
if (name == "uint8") return CV_8U;
if (name == "int8") return CV_8S;
if (name == "uint16") return CV_16U;
if (name == "int16") return CV_16S;
if (name == "uint32") return CV_32U;
if (name == "int32") return CV_32S;
if (name == "uint64") return CV_64U;
if (name == "int64") return CV_64S;
return -1;
}
// element-wise op function name -> (op, arity), or arity 0 if unknown
static TOp fnOp(const std::string& name, int& arity)
{
struct E { const char* n; TOp op; };
static const E unary[] = { {"abs",OP_ABS},{"sqrt",OP_SQRT},{"exp",OP_EXP},{"log",OP_LOG},
{"sin",OP_SIN},{"cos",OP_COS},{"tanh",OP_TANH},{"erf",OP_ERF},
{"relu",OP_RELU} };
static const E binary[] = { {"max",OP_MAX},{"min",OP_MIN},{"pow",OP_POW},{"absdiff",OP_ABSDIFF},
{"hypot",OP_HYPOT},{"mag",OP_HYPOT}, // mag = the cv::magnitude-flavored alias
{"atan2",OP_ATAN2} };
static const E tern[] = { {"clamp",OP_CLAMP},{"select",OP_SELECT} };
for (const E& e : unary) if (name == e.n) { arity = 1; return e.op; }
for (const E& e : binary) if (name == e.n) { arity = 2; return e.op; }
for (const E& e : tern) if (name == e.n) { arity = 3; return e.op; }
arity = 0; return OP_NOP;
}
// --- parser ------------------------------------------------------------------------------
// Builds the program DIRECTLY into the TExpr (no intermediate graph), exactly like the hand
// builders: each parse step calls e.emitUnary()/emitBinary()/addConst()/output() and returns the arg
// SLOT holding its value. Input depths are known up front (from the input Mats), so every operand is
// typed as it is parsed - the emit* layers infer result depths and insert casts on the spot.
struct Parser
{
Lexer lex;
Token cur;
TExpr& e;
const int* inputSlot; // slot id of each external input (precreated)
int ninputs;
std::map<std::string, int> env; // named temporaries -> slot
Parser(std::string_view src, TExpr& expr, const int* islot, int nin)
: lex(src), e(expr), inputSlot(islot), ninputs(nin) { cur = lex.next(); }
void advance() { cur = lex.next(); }
bool isOp(const char* op) const { return cur.type == T_OP && cur.text == op; }
void expect(TokType t, const char* what)
{
CV_Assert(cur.type == t && what);
advance();
}
// A flexible CONST slot holds a literal whose depth the emit* layers pick per use.
bool isFlexConst(int slot) const
{
return e.arginfo[slot].kind == TExpr::CONST && e.arginfo[slot].depth == EW_DEPTH_NONE;
}
int parsePrimary()
{
if (cur.type == T_NUM) { int s = e.addConst(EW_DEPTH_NONE, Scalar(cur.num), 1); advance(); return s; }
if (cur.type == T_INPUT) { int idx = cur.input; advance();
CV_Assert(idx >= 0 && idx < ninputs && "ew::expression: input index out of range");
return inputSlot[idx]; }
if (cur.type == T_LPAREN){ advance(); int x = parseTernary(); expect(T_RPAREN, "expected ')'"); return x; }
if (cur.type == T_IDENT)
{
std::string name = cur.text; advance();
if (cur.type != T_LPAREN) // variable reference
{
auto it = env.find(name);
CV_Assert(it != env.end() && "ew::expression: undefined name");
return it->second;
}
advance(); // consume '('
// every function takes <= 3 args (clamp/select); a stack buffer avoids a std::vector
// heap alloc. The {} is for gcc's -Wmaybe-uninitialized only.
std::array<int, 8> args = {}; int nargs = 0;
if (cur.type != T_RPAREN)
{
args[nargs++] = parseTernary();
while (cur.type == T_COMMA) { advance();
CV_Assert(nargs < (int)args.size() && "ew::expression: too many arguments");
args[nargs++] = parseTernary(); }
}
expect(T_RPAREN, "expected ')'");
int td = typeDepth(name);
if (td >= 0)
{
CV_Assert(nargs == 1); // type cast: just convert the operand
int s = args[0];
return isFlexConst(s) ? e.typedConstFrom(s, td)
: e.maybeAddCast(s, td);
}
int arity = 0; TOp op = fnOp(name, arity);
CV_Assert(arity != 0 && "ew::expression: unknown function");
CV_Assert(nargs == arity && "ew::expression: wrong number of arguments");
if (arity == 1) return e.emitUnary(op, args[0]);
if (arity == 2) return e.emitBinary(op, args[0], args[1]);
return e.emitTernary(op, args[0], args[1], args[2]);
}
CV_Error(Error::StsParseError, "ew::expression: expected a primary expression");
}
int parseUnary()
{
if (isOp("-") || isOp("!"))
{
std::string op = cur.text; advance();
int operand = parseUnary();
// constant-fold a leading sign so that "-1.5" does not need an OP_NEG kernel
if (op == "-" && isFlexConst(operand))
{
AutoBuffer<double, MAX_LOCAL_CN> v; int cn = constDoubles(e, operand, v);
Scalar neg; for (int i = 0; i < cn && i < 4; i++) neg[i] = -v[i];
return e.addConst(EW_DEPTH_NONE, neg, std::min(cn, 4));
}
return e.emitUnary(op == "-" ? OP_NEG : OP_NOT, operand);
}
return parsePrimary();
}
int parseExpr(int minPrec)
{
int left = parseUnary();
while (cur.type == T_OP)
{
int p = binPrec(cur.text);
if (p < minPrec) break;
std::string op = cur.text; advance();
// '**' is right-associative (a ** b ** c == a ** (b ** c)): recurse at the SAME
// precedence so the right side swallows further '**'s; everything else at p+1.
int right = parseExpr(op == "**" ? p : p + 1);
left = e.emitBinary(binOp(op), left, right);
}
return left;
}
// The ternary conditional cond ? a : b == select(cond, a, b). Lowest precedence (below all
// arithmetic/compare/bitwise), right-associative: f1 ? a : f2 ? b : c groups as
// f1 ? a : (f2 ? b : c). This is the general expression entry point.
int parseTernary()
{
int cond = parseExpr(0);
if (!isOp("?")) return cond;
advance();
int thenv = parseTernary();
CV_Assert(isOp(":") && "ew::expression: expected ':' in a ?: conditional");
advance();
int elsev = parseTernary(); // right-associative chain
return e.emitTernary(OP_SELECT, cond, thenv, elsev);
}
// final result: a single expression, or a top-level tuple (e0, e1, ...).
void parseResult()
{
if (cur.type == T_LPAREN)
{
// backtrack point: save the lexer AND the program length, since the trial parse below
// emits instructions/slots that must be rolled back if this turns out NOT to be a tuple.
Lexer save = lex; Token savedCur = cur;
size_t nInsn = e.prog.size(), nArg = e.arginfo.size(); int nTemp = e.ntemps;
advance();
int e0 = parseTernary();
if (cur.type == T_COMMA)
{
e.output(e0);
while (cur.type == T_COMMA) { advance(); e.output(parseTernary()); }
expect(T_RPAREN, "expected ')'");
return;
}
lex = save; cur = savedCur; // not a tuple -> undo the trial parse
e.prog.resize(nInsn); e.arginfo.resize(nArg); e.ntemps = nTemp;
}
e.output(parseTernary());
}
void parse()
{
while (true)
{
if (cur.type == T_IDENT)
{
Lexer save = lex; Token savedCur = cur; // lexer-only backtrack (nothing emitted yet)
std::string name = cur.text; advance();
if (cur.type == T_ASSIGN)
{
advance();
env[name] = parseTernary();
expect(T_SEMI, "expected ';' after assignment");
continue;
}
lex = save; cur = savedCur; // not an assignment
}
parseResult();
if (cur.type == T_SEMI) advance();
break;
}
CV_Assert(cur.type == T_END && "ew::expression: trailing tokens");
CV_Assert(e.noutputs > 0 && "ew::expression: no result");
}
};
} // anonymous namespace
}} // namespace cv::ew
namespace cv {
// Public entry point (declared in opencv2/core.hpp): parse a broadcasting element-wise expression,
// compile it and run it over the inputs. This IS the engine's string front-end - there is no
// separate cv::ew::expression indirection.
void texpr(const std::string& expr, InputArrayOfArrays _inputs, OutputArrayOfArrays _outputs)
{
using namespace cv::ew;
CV_INSTRUMENT_REGION();
CV_Assert(_inputs.kind() == _InputArray::STD_VECTOR_MAT);
const std::vector<Mat>& inps = *(const std::vector<Mat>*)_inputs.getObj();
const int ninputs = (int)inps.size();
// Input depths are known now, so build a fully-typed program straight away: one INPUT slot per
// input (slot index i carries input i's depth), then parse the expression into instructions.
TExpr e;
AutoBuffer<int> islot(std::max(ninputs, 1));
for (int i = 0; i < ninputs; i++) islot[i] = e.addInput(inps[i].depth());
Parser(expr, e, islot.data(), ninputs).parse();
e.compile();
auto kind = _outputs.kind();
if (kind == _InputArray::STD_VECTOR_MAT) {
std::vector<Mat>& outs = _outputs.getMatVecRef();
outs.resize(e.noutputs);
e.exec(inps.data(), outs.data());
} else {
CV_Error(Error::StsNotImplemented, "vector<Mat> is expected as output of texpr");
}
}
}
modules/core/src/arithm_expr.hpp
+346
View File
@@ -0,0 +1,346 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// The new element-wise expression engine - low-level contract shared by the kernels
// (arithm.simd.hpp), the per-op dispatchers (arithm.dispatch.cpp) and the
// graph compiler / executor / parser (arithm_expr.cpp).
//
// Private core header for now; once cv::add() is rebuilt on top of it, the public-facing parts
// (cv::expression, the get*Func entry points) move to external headers. Assumes precomp.hpp (Mat,
// MatShape, AutoBuffer, Scalar) is already included.
//
// Design notes (agreed):
// - Universal arity: every instruction is {fptr, arg0, arg1, arg2, result}; unused operands
// reference the reserved "none" arg slot (index 0).
// - A kernel processes one 2D tile of a single slice; broadcasting is per-operand y/x steps
// (step 0 = re-read). Steps are in ELEMENTS; dst is contiguous in x (dst.stepx == 1).
#ifndef OPENCV_CORE_ARITHM_EXPR_HPP
#define OPENCV_CORE_ARITHM_EXPR_HPP
#include <array>
#include <iosfwd>
#include <string_view>
#include <utility>
#include <vector>
namespace cv { namespace ew {
// Sentinel depth for an unused operand (note: CV_8U == 0 is a *valid* depth,
// so the "no operand" marker must be negative).
enum { EW_DEPTH_NONE = -1 };
// Each op carries a fixed numerical value whose high bits encode its arity, so the arity
// can be recovered with a shift and no lookup table: arity = (op >> OP_ARITY_SHIFT) & 7.
// The ops are grouped into contiguous arity blocks (unary = 1<<10, binary = 2<<10,
// ternary = 3<<10); within a block the low bits are just a running index.
enum { OP_ARITY_SHIFT = 10 };
enum
{
OP_UNARY_BASE = 1 << OP_ARITY_SHIFT, // 0x400
OP_BINARY_BASE = 2 << OP_ARITY_SHIFT, // 0x800
OP_TERNARY_BASE = 3 << OP_ARITY_SHIFT // 0xC00
};
// The single enumeration of element-wise operations, used both in the IR and by the
// kernel dispatcher.
enum TOp
{
OP_NOP = 0,
// ---------------- unary (arity 1) ----------------
OP_NEG = OP_UNARY_BASE, OP_ABS, OP_NOT,
OP_SQRT, OP_EXP, OP_LOG,
OP_SIN, OP_COS, OP_TANH, OP_ERF, OP_RELU,
OP_CAST, // saturating type conversion, no scaling
// ---------------- binary (arity 2) ----------------
OP_ADD = OP_BINARY_BASE, OP_SUB, OP_MUL, OP_DIV, OP_POW,
OP_MIN, OP_MAX, OP_ABSDIFF,
OP_AND, OP_OR, OP_XOR,
// compare -> mask (result depth given explicitly, e.g. CV_Bool/CV_8U)
OP_CMP_EQ, OP_CMP_NE, OP_CMP_LT, OP_CMP_LE, OP_CMP_GT, OP_CMP_GE,
// addWeighted: a*alpha + b*beta + gamma (params = {alpha, beta, gamma}). A fused composite, not a
// kernel - emitBinary expands it. Placed last in the binary group so it doesn't renumber the ops
// above it (some dispatch is by enum value).
OP_ADDW,
// hypot(x, y) = sqrt(x^2 + y^2), the magnitude of a 2D vector. NAIVE evaluation (matching
// cv::magnitude), NOT the overflow-safe std::hypot. Computes in the float domain like OP_POW.
OP_HYPOT,
// atan2(y, x) in RADIANS, standard C range (-pi, pi] (NB: cv::phase/fastAtan2 use degrees
// [0, 360)). arg0 = y, arg1 = x, like std::atan2. Float domain, like OP_POW/OP_HYPOT.
OP_ATAN2,
// ---------------- ternary (arity 3) ----------------
OP_CLAMP = OP_TERNARY_BASE, // clamp(x, lo, hi)
// select(mask, a, b) (a.k.a. where): dst = (mask != 0) ? a : b; mask is 1 byte (bool/u8/s8),
// never cast. Also the engine's masked-op tail: cv::add(..., mask) computes into a temp r,
// then select(mask, r, dst) -> dst overwrites only the masked subset of the (pre-existing)
// output (dst rides as both arg2 and the result slot; the kernel is alias-safe).
OP_SELECT,
OP_CONVERT_SCALE // cast<rdepth>(src*scale + offset); scale/offset may be tensors
};
// Arity from the encoding above (0 for OP_NOP). No table to keep in lock-step with the enum.
inline int opArity(TOp op) { return ((int)op >> OP_ARITY_SHIFT) & 7; }
// Operation category — the graph compiler's type-inference rules differ per category.
enum ElemwiseCategory
{
CAT_ARITH = 0, // numeric, result type follows promotion rules
CAT_BITWISE, // integer-only, same-type
CAT_COMPARE, // produces a mask, result depth is explicit
CAT_MATH, // transcendental, float domain
CAT_CAST, // type conversion (with/without scaling)
CAT_SELECT // data-routing (select/where)
};
// Op metadata (implemented in arithm_expr.cpp).
CV_EXPORTS const char* opName(TOp op);
CV_EXPORTS ElemwiseCategory opCategory(TOp op);
// numpy-style arithmetic promotion of two depths (see arithm_expr.cpp): INTEGER-PRESERVING and
// COMMUTATIVE; mixed sign -> a wide-enough signed type; any float -> the smallest covering float.
// EW_DEPTH_NONE on one side returns the other. The rdepth==-1 auto result-depth rule.
CV_EXPORTS int promoteArith(int a, int b);
// The 'safe' result depth of absdiff over a value of `depth`: a SIGNED integer difference can reach
// 2^width-1 (|(-128)-127| = 255), so it needs the UNSIGNED type of the same width (8s->8u, 16s->16u,
// 32s->32u, 64s->64u) to hold it without saturation. Unsigned/float depths keep their type.
CV_EXPORTS int absdiffResultDepth(int depth);
// ---------------------------------------------------------------------------
// Steps for one operand, in elemsize1 units, one entry per shape dim (parallel to a
// MatShape). A 0 entry means broadcast along that axis. Heap-free, like MatShape.
// ---------------------------------------------------------------------------
typedef std::array<size_t, MatShape::MAX_DIMS> EwSteps;
// ---------------------------------------------------------------------------
// The low-level kernel contract.
//
// Processes a width x height tile of one slice. Each source operand carries its own
// (stepy, stepx) in elements; a 0 step means broadcast along that axis. stepx is restricted
// to {0,1} (1 = contiguous, 0 = broadcast-scalar along x): the general strided/gather case is
// excluded and the executor guarantees the invariant. The result is contiguous in x (dst stepx == 1).
// Returns >= 0 on success, < 0 (a CV_HAL_ERROR_* code) to let the caller fall back.
//
// The trailing `params` points at the instruction's scalar parameter block (Insn::params, a cv::Scalar's
// 4 doubles): mul/div read params[0] as a scale (1.0 = none). `flags` carries small per-kernel options
// (e.g. a compare op's 0/1-vs-0/255 mask value); `userdata` carries a wrapped core BinaryFunc for casts.
// ---------------------------------------------------------------------------
typedef int (*KernelFunc)(
const void* src0, size_t step0y, size_t step0x,
const void* src1, size_t step1y, size_t step1x,
const void* src2, size_t step2y, size_t step2x,
void* dst, size_t dstepy,
int width, int height, const double* params,
int flags, void* userdata);
// TKernel::flags - small per-kernel options, interpreted by the kernel itself (so the meaning is
// per-kernel: the cast kernels read it as the dst element size, the compare kernels as the bits below).
enum TKernelFlags
{
EW_KERNEL_MASK1 = 1, // compare: emit a 0/1 mask instead of the default 0/255 (cv::compare)
EW_KERNEL_SWAP01 = 2, // compare: the kernel swaps its own src0<->src1 (with their steps), so
// LT/LE reuse the GT/GE kernels (a<b == b>a, a<=b == b>=a)
// compare fuses its post-op fix-up: the u8 result is (rawmask & M) | V per channel. Uniform (this
// bit clear): M = trueVal (MASK1 ? 1 : 255), V = 0 - an ordinary compare. Per-channel (this bit
// set, only the divergent multi-channel scalar case): M/V come from the 4-bit fields below. This
// folds the former separate patch pass into the compare kernel (one pass, no extra kernel).
EW_CMP_PATCH = 4,
EW_CMP_PATCH_SHIFT = 8, // per channel c in [0,4): bits [SHIFT+c*4 .. +4) = 2 bits M then 2 bits V;
// each 2-bit field decodes 0->0x00, 1->0x01, 2|3->0xFF.
};
// Decode a 2-bit patch field (0->0, 1->1, else 255) - shared by the compare kernel and its builder.
inline int cmpPatchByte(int twoBits) { return twoBits == 0 ? 0 : twoBits == 1 ? 1 : 255; }
struct TKernel
{
KernelFunc fptr = nullptr;
void* userdata = nullptr;
int flags = 0;
};
// ---- per-op kernel entry points (implemented in arithm.dispatch.cpp) ----------------------
// Each returns the kernel optimized for the current CPU (it forwards through CV_CPU_DISPATCH to the
// matching get*Func_ compiled per SIMD baseline in arithm.simd.hpp). `T` is the (common) input
// depth, `R` the result depth; EW_DEPTH_NONE marks an unused operand. A null fptr means "no exact
// kernel for this combination" - the compiler then inserts OP_CAST and retries with a working type.
// These are the useful, op-specific intermediaries (candidates for CV_EXPORTS later).
CV_EXPORTS TKernel getAddFunc(int T, int R);
CV_EXPORTS TKernel getSubFunc(int T, int R);
CV_EXPORTS TKernel getMulFunc(int T, int R);
CV_EXPORTS TKernel getDivFunc(int T, int R, bool checked); // `checked` => guard b==0 -> 0 (integer divide)
CV_EXPORTS TKernel getPowFunc(int T, int R);
CV_EXPORTS TKernel getMinFunc(int T, int R);
CV_EXPORTS TKernel getMaxFunc(int T, int R);
CV_EXPORTS TKernel getAbsdiffFunc(int T, int R);
CV_EXPORTS TKernel getHypotFunc(int T, int R); // OP_HYPOT, T x T -> T (float depths)
CV_EXPORTS TKernel getAtan2Func(int T, int R); // OP_ATAN2, T x T -> T (float depths)
CV_EXPORTS TKernel getCmpFunc(TOp op, int T); // T x T -> u8 mask (op = OP_CMP_*)
CV_EXPORTS TKernel getBitwiseFunc(TOp op, int esz); // OP_AND / OP_OR / OP_XOR, by element size
CV_EXPORTS TKernel getNotFunc(int esz); // OP_NOT, by element size
CV_EXPORTS TKernel getAddWeightedFunc(int T, int R); // OP_ADDW, a*alpha+b*beta+gamma (T x T -> R)
CV_EXPORTS TKernel getSelectFunc(int mdepth, int T); // OP_SELECT: 1-byte mask, a/b/dst of T
CV_EXPORTS TKernel getClampFunc(int T); // OP_CLAMP: min(max(x, lo), hi), all of T
// math.dispatch.cpp (kernels in math.simd.hpp):
CV_EXPORTS TKernel getMathFunc(TOp op, int T); // unary math (OP_SQRT..OP_RELU), T -> T,
// T in {f16, bf16, f32, f64}; exp/log at
// f32/f64 route through HAL/IPP when installed
// the engine's OWN vector kernel over one contiguous span - the built-in implementation behind
// cv::hal::exp32f & co (their table kernels are gone), and getMathFunc's final fallback
CV_EXPORTS void mathSpanEngine(TOp op, int depth, const void* src, void* dst, int n);
// getPowFunc is declared above with the arithm getters but LIVES in math.dispatch.cpp too
// (powKernel: special-cased scalar exponents + the exp(y*log(x)) general path)
// The op-level dispatcher: routes (op, depths) to the right get*Func above. nullptr if the exact
// combination is not provided. Unused operand depths are EW_DEPTH_NONE.
CV_EXPORTS TKernel getElemwiseFunc(TOp op, int depth0, int depth1, int depth2, int rdepth);
// An element-wise expression as ONE flat program (the analogue of cv::MatExpr for element-wise ops):
// an arg table (`arginfo`, the typed operands) + an instruction list (`prog`). The program IS the
// representation - it is built directly:
// - declare operands with addInput()/addConst()/addOutput() (and addTemp() for intermediates);
// - append operations with addInsn() (a single op, you pick the slots) or, for automatic type
// inference + cast insertion, with emitUnary()/emitBinary()/emitTernary().
// Operand types are known at build time, so addInsn resolves each instruction's kernel on the spot;
// compile() is a cheap finalize pass (pack temps into physical buffers via liveness).
//
// Heap-free for the common case: to back cv::add() the program is (re)built every call, so the
// containers must not allocate for typical (small) expressions. AutoBuffer keeps a handful of
// insns/slots inline on the stack and only spills to the heap for large expressions; it is copyable,
// so TExpr is still returned/passed by value. A default-constructed AutoBuffer is empty (size()==0)
// and grows like std::vector (push_back amortized 1.5x); clear() resets the size to 0.
struct CV_EXPORTS TExpr
{
// Static (shape-independent) classification of an arg slot.
enum ArgKind
{
NONE = 0, // the reserved empty operand (slot 0)
INPUT,
CONST,
TEMP,
OUTPUT
};
struct Arg
{
ArgKind kind = NONE;
int depth = EW_DEPTH_NONE; // EW_DEPTH_NONE on a CONST = "flexible" (the emit* layers type it per use)
int channels = 0; // for CONST: # of per-channel values (0/1 => single broadcast value)
int index = -1; // input#/output#/temp-id depending on kind
// CONST only: the constant's values live in TExpr::constbuf (in `srcdepth` until compile(),
// which converts them to the resolved `depth`). constofs = offset into constbuf, in uint64_t
// units. A per-channel scalar of any width is carried this way (no 4-channel Scalar limit).
int srcdepth = EW_DEPTH_NONE;
size_t constofs = 0;
};
// One compiled instruction: the op + arg-table indices + a resolved kernel (TKernel: fptr +
// userdata + flags). Every kernel uses the SAME calling convention (the universal KernelFunc);
// OP_CAST / OP_CONVERT_SCALE bind castKernel, which carries the core convert BinaryFunc in
// kernel.userdata. `params` is the per-instruction scalar block (mul/div scale in params[0];
// convert_scale {scale, offset} in params[0..1]). The kernel is bound by addInsn at build time.
struct Insn
{
TKernel kernel;
TOp op = OP_NOP;
int arg0 = 0, arg1 = 0, arg2 = 0, result = 0;
Scalar params = Scalar(1); // op scalars; params[0]=scale defaults to 1 (identity)
};
AutoBuffer<Insn, 16> prog; // instructions, in execution order (kernels bound by addInsn)
AutoBuffer<Arg, 16> arginfo; // slot 0 is always NONE
AutoBuffer<uint64_t, 16> constbuf; // CONST value store (uint64-aligned slots); Arg::constofs
// indexes it. Source values on build, resolved-type after compile()
int ninputs = 0;
int noutputs = 0;
int ntemps = 0;
int nbuffers = 0; // distinct physical temp buffers after liveness
int nconsts = 0; // # materialized CONST slots (set by compile()) - sizes
// the const store; nconsts==0 enables exec()'s fast path
int capElems = 0; // # elements one ~16KB L1 scratch fragment holds (set by
// compile()); INT_MAX when there are no temps.
AutoBuffer<int, 16> bufferOfTemp; // temp-id -> physical buffer id
AutoBuffer<int, 8> bufEszPrefix; // [nbuffers+1] prefix sums of each physical temp buffer's
// elem size (set by compile()); [nbuffers] = temp bytes
// per output element.
TExpr();
void clear(); // reset to an empty program (slot 0 = NONE)
void dump(std::ostream& os) const; // human-readable slot table + instruction list (debug)
// ---- operand / instruction builders (return the new slot / instruction index) ----
int addInput(int depth);
int addConst(int depth, const Scalar& v, int channels = 1); // source = Scalar (f64); depth NONE => flexible
int addConst(int depth, int srcdepth, const void* data, int channels); // source = native bytes
// A typed copy of a flexible CONST `srcSlot` at the resolved `depth` (shares its source values;
// compile() converts them). Used by the emit* layers / parser cast where addConst(depth, cval) was.
int typedConstFrom(int srcSlot, int depth);
int addTemp(int depth);
int addOutput(int depth);
// addInsn resolves the instruction's kernel NOW from the operand/result depths (final at build
// time). The 2nd form takes a pre-resolved kernel, for callers that already probed getElemwiseFunc.
int addInsn(TOp op, int a0, int a1, int a2, int result, const Scalar& params = Scalar(1));
int addInsn(TOp op, int a0, int a1, int a2, int result, const TKernel& kernel,
const Scalar& params = Scalar(1));
// Return `arg` unchanged if it is already of depth `depth`; otherwise append an OP_CAST into a
// fresh temp of that depth and return the temp's slot. The one place casts are inserted.
int maybeAddCast(int arg, int depth);
// ---- type-inference + cast-insertion policy layers (parser + hand builders) ----
// Each derives the result depth (promotion, or a forced `rdepth`), picks the compute depth and a
// wide fallback per op family, materializes flexible CONST operands, casts every operand to the
// compute depth, then emits `op` (direct when a kernel exists, else compute wide and cast down).
// Returns the slot holding the result. `rdepth` EW_DEPTH_NONE = auto.
int emitUnary(TOp op, int a, int rdepth = EW_DEPTH_NONE, const Scalar& params = Scalar(1));
int emitBinary(TOp op, int a, int b, int rdepth = EW_DEPTH_NONE, const Scalar& params = Scalar(1));
int emitTernary(TOp op, int a, int b, int c, int rdepth = EW_DEPTH_NONE);
// Land `temp` in the existing slot `out`: redirect temp's single producer to write `out` directly
// (dropping a dead last temp so compile() keeps its no-temp fast exit); otherwise copy via OP_CAST.
int moveToOutput(int temp, int out);
// Declare a result tensor fed by `rootSlot` (a fresh OUTPUT of its depth) and moveToOutput into it.
int output(int rootSlot);
// Finalize: pack temps into physical buffers (liveness), count consts, size the L1 fragment cap.
void compile();
// The broadcast output geometry for the given inputs (spatial dims + channel count, channels
// innermost). All outputs share it; their depth comes from arginfo. Lets a caller pre-create the
// destination (dst.create(spatial, CV_MAKETYPE(depth, channels))) before exec writes into it.
// Inputs are passed as an array of pointers (no Mat-header copies in the hot path).
void outputShape(const Mat* const* inputs, MatShape& spatial, int& channels) const;
// Execute the compiled program over a set of input Mats, producing the broadcast result(s). If an
// output Mat already has the right shape/type it is reused (not reallocated). Inputs are passed as
// an array of pointers.
void exec(const Mat* const* inputs, Mat* outputs);
// Convenience overloads: inputs as a contiguous array of Mats (builds the pointer array + forwards).
// Handy for callers holding a Mat[]/vector<Mat>; the hot path should pass pointers directly.
void outputShape(const Mat* inputs, MatShape& spatial, int& channels) const;
void exec(const Mat* inputs, Mat* outputs);
};
// ---- hand builders (the stand-ins the future engine-backed cv::add etc. are built on) ----
// Compose a binary op (ADD/SUB/MUL/DIV/MIN/MAX/ABSDIFF/CMP_*) for any (depth0, depth1, rdepth):
// cast operands to a common type, op direct-or-wide-then-cast. maskDepth != EW_DEPTH_NONE adds a
// write-mask input (#2); scale != 1 (mul/div) rides the instruction's params[0].
CV_EXPORTS void makeBinaryArithProgram(TExpr& p, TOp op, int depth0, int depth1, int rdepth,
int maskDepth = EW_DEPTH_NONE, double scale = 1.0);
// addWeighted(a,alpha,b,beta,gamma) = a*alpha + b*beta + gamma (two fused convert_scale MACs + add).
CV_EXPORTS void makeAddWeightedProgram(TExpr& p, int depth0, int depth1, int rdepth,
double alpha, double beta, double gamma);
// NOTE: the string front-end is the PUBLIC cv::texpr() (declared in opencv2/core.hpp, defined in
// arithm_expr.cpp) - there is no cv::ew::expression() indirection.
}} // namespace cv::ew
#endif // OPENCV_CORE_ARITHM_EXPR_HPP
modules/core/src/broadcast.cpp
+401
View File
@@ -0,0 +1,401 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// Implementation of cv::BroadcastOp (declared in opencv2/core/mat.hpp): an op-agnostic driver for
// broadcasting element-wise traversal. It computes the numpy-broadcast iteration space over a flat
// list of operand Mats, collapses contiguous dims, partitions into tiles for parallel_for_, and hands
// each 2D tile's per-operand slices to a `body` callback (which owns all semantics).
#include "precomp.hpp"
#include <algorithm>
#include <array>
#include <climits>
#include <cmath>
namespace cv {
// Per-operand steps along the (collapsed) iteration axes are kept in a MatStep (a value-type holding
// MAX_DIMS size_t entries). Here the entries are steps in ELEMENTS (one scalar / channel value), not
// the byte steps a Mat stores - the container is reused for its fixed-size storage and [] access.
// ---------------------------------------------------------------------------
// Geometry helpers.
// ---------------------------------------------------------------------------
// How a Mat's channels are mapped into the logical (shape, step, esz1) handed to the geometry:
// CH_FOLD : channels stay scalar-wise, folded into the innermost dim (back() *= cn, step 1).
// Used (expandChannels=true) when no channel broadcast is needed (all single-channel,
// or all same-cn with equal back()) - the body then sees single-channel data.
// CH_DIM : channels become an explicit innermost iteration dim (cn, step 1); single-channel
// operands get a size-1 channel that broadcasts 1->N. Used when channel broadcast is
// needed (mixed channel counts, or multichannel with differing back()).
// CH_ELEM : channels stay inside the element (esz = full elemSize, no channel dim). Used with
// expandChannels=false; the body handles channels itself (deinterleave fast path).
enum ChMode { CH_FOLD, CH_DIM, CH_ELEM };
static void matLayout(const Mat& m, ChMode mode, MatShape& shp, MatStep& step, int& esz1)
{
const int nd = m.dims, cn = m.channels();
if (mode == CH_ELEM)
{
esz1 = (int)m.elemSize(); // one full (cn-channel) pixel
shp.resize(nd);
for (int i = 0; i < nd; i++) { shp[i] = m.size[i]; step[i] = m.step[i] / esz1; }
return;
}
esz1 = (int)m.elemSize1(); // one scalar (channel value)
if (mode == CH_DIM)
{
shp.resize(nd + 1);
for (int i = 0; i < nd; i++) { shp[i] = m.size[i]; step[i] = m.step[i] / esz1; }
shp[nd] = cn; step[nd] = 1; // channels = explicit innermost dim
}
else // CH_FOLD
{
if (nd == 0) // 0-dim scalar: channels are the only dim
{
shp.assign(1, cn);
step[0] = 1;
}
else
{
shp.resize(nd);
for (int i = 0; i < nd; i++) { shp[i] = m.size[i]; step[i] = m.step[i] / esz1; }
shp[nd - 1] *= cn; // fold channels into the innermost dim
step[nd - 1] = 1; // scalars are contiguous there
}
}
}
// A single-channel scalar (one value, cn==1, total()==1) broadcasts into everything trivially
// (step 0 on every axis incl. channels), so it must NOT force CH_DIM - it is excluded from the
// channel-mode decision entirely.
static bool isSingleChannelScalar(const Mat& m)
{
return m.channels() == 1 && m.total() == 1;
}
// Decide (globally, across all operands) how channels are presented for expandChannels=true:
// CH_FOLD when no channel broadcast is needed, CH_DIM when it is. Single-channel scalars are
// excluded first; among the rest, multichannel operands must all share the same cn (an (n,m) mix
// with both > 1 is an error).
static ChMode decideChannelMode(const Mat* const* arrays, int K)
{
int N = 1; // the single multichannel count, if any
for (int k = 0; k < K; k++)
{
if (isSingleChannelScalar(*arrays[k])) continue;
int c = arrays[k]->channels();
if (c > 1) { if (N == 1) N = c; else CV_Assert(N == c && "ew: (n,m) channel mix unsupported"); }
}
if (N == 1) return CH_FOLD; // all single-channel -> fold (a no-op)
bool allMulti = true, sameBack = true;
int back = -1;
for (int k = 0; k < K; k++)
{
const Mat& a = *arrays[k];
if (isSingleChannelScalar(a)) continue;
if (a.channels() != N) allMulti = false;
int b = a.dims > 0 ? a.size[a.dims - 1] : 1; // 0-dim scalar has no spatial back (=1)
if (back < 0) back = b; else if (b != back) sameBack = false;
}
return (allMulti && sameBack) ? CH_FOLD : CH_DIM; // fold only if no channel broadcast
}
// numpy-style broadcast of several right-aligned shapes.
static bool broadcastShape(const MatShape* shps, int K, MatShape& out)
{
size_t nd = 0;
for (int k = 0; k < K; k++) nd = std::max(nd, shps[k].size());
out.assign(nd, 1);
for (int k = 0; k < K; k++)
{
const MatShape& s = shps[k];
size_t off = nd - s.size();
for (size_t i = 0; i < s.size(); i++)
{
int d = s[i], &o = out[off + i];
if (o == 1) o = d;
else if (d != 1 && d != o) return false;
}
}
return true;
}
// Right-align an arg's own (shp,step) to nd dims; broadcast dims get step 0.
static void alignArg(const MatShape& shp, const MatStep& step, int nd,
MatStep& as, MatShape& ash)
{
as.clear();
ash.assign(nd, 1);
int off = nd - (int)shp.size();
for (int i = 0; i < (int)shp.size(); i++)
{
int d = shp[i];
ash[off + i] = d;
as[off + i] = (d == 1) ? 0 : step[i];
}
}
// Collapse adjacent dims that are contiguous (and broadcast-consistent) across all args.
static int collapseDims(MatStep* S, MatShape* H, int K, MatShape& D)
{
int nd = (int)D.size();
if (nd <= 1) return nd;
int j = nd - 1;
for (int i = j - 1; i >= 0; i--)
{
bool contig = true, scalar = true, consist = true;
for (int k = 0; k < K; k++)
{
size_t st = S[k][j] * (size_t)H[k][j];
bool prevScalar = H[k][j] == 1;
bool curScalar = H[k][i] == 1;
contig = contig && (st == S[k][i]);
scalar = scalar && curScalar;
consist = consist && (curScalar == prevScalar);
}
if (contig && (consist || scalar))
{
for (int k = 0; k < K; k++) H[k][j] *= H[k][i];
D[j] *= D[i];
}
else
{
j--;
if (i < j)
{
for (int k = 0; k < K; k++) { H[k][j] = H[k][i]; S[k][j] = S[k][i]; }
D[j] = D[i];
}
}
}
int m = nd - j;
for (int d = 0; d < m; d++)
{
D[d] = D[j + d];
for (int k = 0; k < K; k++) { S[k][d] = S[k][j + d]; H[k][d] = H[k][j + d]; }
}
D.resize(m);
for (int k = 0; k < K; k++) H[k].resize(m);
// Zero out steps of broadcast (size-1) dims (numpy step==0 trick).
for (int d = 0; d < m; d++)
for (int k = 0; k < K; k++)
if (H[k][d] == 1) S[k][d] = 0;
return m;
}
// Fast geometry for the dominant case: every operand is either (a) an array sharing ONE common
// shape - same dims, sizes and channel count - and contiguous, or (b) a single-channel scalar
// (cn==1, total()==1). Then the whole traversal is a single contiguous 1D run of `total` scalars
// (channels folded in): arrays get stepx 1, scalars stepx 0. This skips decideChannelMode /
// broadcastShape / alignArg / collapseDims and all their per-operand buffers entirely. Returns
// false (leaving outputs untouched) when the operands don't fit, so the caller runs general
// geometry. expandChannels=false (CH_ELEM) keeps channels in the element and is left to general.
static bool fastSameShape(const Mat* const* arrays, int K, bool expandChannels,
uchar** base, int* esz1, MatStep* S, MatShape& D, int& m)
{
if (!expandChannels) return false;
int ref = -1;
for (int k = 0; k < K; k++)
if (!isSingleChannelScalar(*arrays[k])) { ref = k; break; }
if (ref < 0) return false; // all single-channel scalars: let general handle
const Mat& R = *arrays[ref];
const int rdims = R.dims, rcn = R.channels();
for (int k = 0; k < K; k++)
{
const Mat& a = *arrays[k];
if (isSingleChannelScalar(a)) continue;
if (a.channels() != rcn || a.dims != rdims || !a.isContinuous()) return false;
for (int i = 0; i < rdims; i++) if (a.size[i] != R.size[i]) return false;
}
const long long total = (long long)R.total() * rcn; // channels folded into the 1D run
CV_Assert(total <= (long long)INT_MAX);
for (int k = 0; k < K; k++)
{
const Mat& a = *arrays[k];
base[k] = (uchar*)a.data;
esz1[k] = (int)a.elemSize1();
S[k][0] = isSingleChannelScalar(a) ? 0 : 1;
}
D.assign(1, (int)total);
m = 1;
return true;
}
// ---------------------------------------------------------------------------
// BroadcastOp::run
// ---------------------------------------------------------------------------
// At namespace scope, NOT inside run(): MSVC 2019 loses the constexpr-ness of function-local
// constants used as template arguments inside a lambda (C2975).
static constexpr int MAX_DIMS = MatShape::MAX_DIMS;
static constexpr int LOCAL_OPS = 8;
void BroadcastOp::run(const Mat* const* arrays, int narrays,
const std::function<void(const Tile&)>& body,
bool expandChannels,
double nstripes)
{
const int K = narrays;
CV_Assert(K >= 1 && arrays != nullptr);
// ---- 1-3. geometry: per-operand collapsed steps S[k], element sizes esz1[k], base
// pointers, and the collapsed iteration shape D (m dims). The fast path handles the
// dominant "all same-shape arrays (+ single-channel scalars)" case in one shot; the
// general path does decideChannelMode + broadcastShape + align + collapse. ----
AutoBuffer<MatStep, LOCAL_OPS> S(K);
AutoBuffer<int, LOCAL_OPS> esz1(K);
AutoBuffer<uchar*, LOCAL_OPS> base(K);
MatShape D;
int m;
if (!fastSameShape(arrays, K, expandChannels, base.data(), esz1.data(), S.data(), D, m))
{
const ChMode mode = expandChannels ? decideChannelMode(arrays, K) : CH_ELEM;
AutoBuffer<MatShape, LOCAL_OPS> shp(K);
AutoBuffer<MatStep, LOCAL_OPS> stp(K);
for (int k = 0; k < K; k++)
{
matLayout(*arrays[k], mode, shp[k], stp[k], esz1[k]);
base[k] = (uchar*)arrays[k]->data;
}
MatShape full;
CV_Assert(broadcastShape(shp.data(), K, full) && "ew: operands are not broadcast-compatible");
const int nd = (int)full.size();
AutoBuffer<MatShape, LOCAL_OPS> H(K);
for (int k = 0; k < K; k++) alignArg(shp[k], stp[k], nd, S[k], H[k]);
D = full;
m = collapseDims(S.data(), H.data(), K, D);
// For a cv::Mat the innermost (channel/last) axis is contiguous, so after collapse the
// innermost stride is always in {0,1}. No gather, no materialization.
for (int k = 0; k < K; k++)
CV_Assert(S[k][m - 1] <= 1 && "ew: unexpected innermost stride > 1");
}
// ---- 4. inner 2D tile axes: width = D[m-1], height = D[m-2] (if any) ----
const int wAxis = m - 1;
const int hAxis = (m >= 2) ? m - 2 : -1;
const int W = D[wAxis];
const int Hgt = (hAxis >= 0) ? D[hAxis] : 1;
const int nOuter = (hAxis >= 0) ? m - 2 : m - 1; // outer ("plane") axes = [0 .. nOuter)
long long nplanes = 1;
for (int d = 0; d < nOuter; d++) nplanes *= D[d];
// ---- 5. desired parallel stripe count (work hint) ----
const long long total = nplanes * (long long)Hgt * (long long)W;
double stripes = nstripes;
if (stripes <= 0) // broadcastOp can't see the body's cost;
stripes = (double)total * 100.0 / (double)(1 << 18); // assume ~100 cycles/element
const int wantTiles = std::max(1, (int)std::lround(stripes));
// ---- 6. tile only for PARALLELISM. broadcastOp is op-agnostic: it does not know the body's
// temp-buffer footprint, so it does NOT tile for L1 - that is the body's job (it
// re-fragments a tile's width into L1-sized chunks for the fused intermediates).
// Start with the largest tile (one 2D block per plane) and split (height first, then
// width) only until there are at least `wantTiles` tiles. Bigger tiles => fewer
// body/decode calls. Width is G-aligned only in the fully-contiguous (1D) case. ----
const int G = 16; // SIMD/cacheline granule
int tw = W, th = Hgt;
auto ntilesOf = [&](int tw_, int th_) {
long long nw = (W + tw_ - 1) / tw_, nh = (Hgt + th_ - 1) / th_;
return nplanes * nh * nw;
};
long long ntiles = ntilesOf(tw, th);
while (ntiles < wantTiles && th > 1) // split height for parallelism
{
th = (th + 1) / 2;
ntiles = ntilesOf(tw, th);
}
while (ntiles < wantTiles && tw > G) // then split width
{
tw = std::max(G, tw / 2);
if (hAxis < 0 && tw > G) tw -= tw % G; // keep width aligned in the 1D case
ntiles = ntilesOf(tw, th);
}
CV_Assert(ntiles <= (long long)INT_MAX);
const int ntilesW = (W + tw - 1) / tw;
const int ntilesH = (Hgt + th - 1) / th;
// ---- 7. execution; decode tile index -> per-operand slices. stepx/stepy are the same for
// every tile, so they are set ONCE; only the per-tile base pointer is recomputed. ----
auto runRange = [&](const Range& r)
{
AutoBuffer<Slice, LOCAL_OPS> slices(K);
// Fast 1D path (m==1: one contiguous axis after collapse, no outer planes, height 1).
// ntilesH==1 and nplanes==1, so the tile index IS the width-tile index - no div/mod, no
// plane multi-index decode, no inner step loop. This is the same-shape / fully-contiguous
// common case.
if (m == 1)
{
for (int k = 0; k < K; k++) { slices[k].stepy = 0; slices[k].stepx = S[k][0]; }
Tile tile;
tile.height = 1; tile.narrays = K; tile.slices = slices.data();
for (int t = r.start; t < r.end; t++)
{
const int wofs = t * tw, ww = std::min(tw, W - wofs);
for (int k = 0; k < K; k++)
slices[k].ptr = base[k] + (size_t)wofs * S[k][0] * (size_t)esz1[k];
tile.width = ww;
body(tile);
}
return;
}
std::array<int, MAX_DIMS> idx;
for (int k = 0; k < K; k++) // steps are tile-independent: set once
{
slices[k].stepy = (hAxis >= 0) ? S[k][hAxis] : 0;
slices[k].stepx = S[k][wAxis];
}
for (int t = r.start; t < r.end; t++)
{
int wt = t % ntilesW;
int rest = t / ntilesW;
int ht = rest % ntilesH;
int plane = rest / ntilesH;
const int wofs = wt * tw, ww = std::min(tw, W - wofs);
const int hofs = ht * th, hh = std::min(th, Hgt - hofs);
int p = plane; // decode plane -> outer multi-index
for (int d = nOuter - 1; d >= 0; d--) {
int dd = D[d];
int np = p / dd;
idx[d] = p - np * dd;
p = np;
}
for (int k = 0; k < K; k++)
{
size_t off = (size_t)wofs * S[k][wAxis];
for (int d = 0; d < nOuter; d++) off += (size_t)idx[d] * S[k][d];
if (hAxis >= 0) off += (size_t)hofs * S[k][hAxis];
slices[k].ptr = base[k] + off * (size_t)esz1[k];
}
Tile tile;
tile.width = ww; tile.height = hh; tile.narrays = K; tile.slices = slices.data();
body(tile);
}
};
// Single tile (small work, wantTiles==1) => run inline and skip the parallel framework
// entirely: its dispatch (std::function wrap + Range machinery + backend hop) is pure
// overhead when there is nothing to parallelize, and dominates small-array latency.
if (ntiles == 1)
runRange(Range(0, 1));
else
parallel_for_(Range(0, (int)ntiles), runRange, stripes);
}
} // namespace cv
modules/core/src/convert.hpp
+26 -16
View File
@@ -140,6 +140,11 @@ static inline void vx_load_pair_as(const ushort* ptr, v_int32& a, v_int32& b)
b = v_reinterpret_as_s32(ub);
}
static inline void vx_load_pair_as(const ushort* ptr, v_uint32& a, v_uint32& b)
{
v_expand(vx_load(ptr), a, b);
}
static inline void vx_load_pair_as(const short* ptr, v_int32& a, v_int32& b)
{
v_expand(vx_load(ptr), a, b);
@@ -428,11 +433,12 @@ static inline void v_store_pair_as(float* ptr, const v_float32& a, const v_float
static inline void v_store_pair_as(unsigned* ptr, const v_float32& a, const v_float32& b)
{
v_int32 z = vx_setzero_s32();
v_int32 ia = v_max(v_round(a), z);
v_int32 ib = v_max(v_round(b), z);
v_store(ptr, v_reinterpret_as_u32(ia));
v_store(ptr + VTraits<v_int32>::vlanes(), v_reinterpret_as_u32(ib));
// v_round(f32) narrows to v_int32, so values in [2^31, 2^32) saturate to INT32_MAX. Scalar for now
// (same class as the f64->{u64,s64,u32} stores above); a proper f32->u32 intrinsic can replace it.
const int n = VTraits<v_float32>::vlanes();
float buf[VTraits<v_float32>::max_nlanes*2];
v_store(buf, a); v_store(buf + n, b);
for (int i = 0; i < 2*n; i++) ptr[i] = saturate_cast<unsigned>(buf[i]);
}
static inline void v_store_pair_as(uchar* ptr, const v_uint32& a, const v_uint32& b)
@@ -657,27 +663,31 @@ static inline void v_store_pair_as(hfloat* ptr, const v_float64& a, const v_floa
v_pack_store(ptr, v);
}
// f64 -> {u64, s64, u32}: no correct vector path yet - v_round(f64,f64) narrows to v_int32, which
// truncates the 64-bit range (u64/s64) and the upper half of u32. Scalar for now (matches the scalar
// tail these functions already fall back to); a proper f64->s64/u64 intrinsic can replace it later.
static inline void v_store_pair_as(uint64_t* ptr, const v_float64& a, const v_float64& b)
{
v_float64 z = vx_setzero_f64();
v_int64 ia, ib;
v_expand(v_round(v_max(a, z), v_max(b, z)), ia, ib);
v_store(ptr, v_reinterpret_as_u64(ia));
v_store(ptr + VTraits<v_uint64>::vlanes(), v_reinterpret_as_u64(ib));
const int n = VTraits<v_float64>::vlanes();
double buf[VTraits<v_float64>::max_nlanes*2];
v_store(buf, a); v_store(buf + n, b);
for (int i = 0; i < 2*n; i++) ptr[i] = saturate_cast<uint64_t>(buf[i]);
}
static inline void v_store_pair_as(int64_t* ptr, const v_float64& a, const v_float64& b)
{
v_int64 ia, ib;
v_expand(v_round(a, b), ia, ib);
v_store(ptr, ia);
v_store(ptr + VTraits<v_uint64>::vlanes(), ib);
const int n = VTraits<v_float64>::vlanes();
double buf[VTraits<v_float64>::max_nlanes*2];
v_store(buf, a); v_store(buf + n, b);
for (int i = 0; i < 2*n; i++) ptr[i] = saturate_cast<int64_t>(buf[i]);
}
static inline void v_store_pair_as(unsigned* ptr, const v_float64& a, const v_float64& b)
{
v_int32 iab = v_max(v_round(a, b), vx_setzero_s32());
v_store(ptr, v_reinterpret_as_u32(iab));
const int n = VTraits<v_float64>::vlanes();
double buf[VTraits<v_float64>::max_nlanes*2];
v_store(buf, a); v_store(buf + n, b);
for (int i = 0; i < 2*n; i++) ptr[i] = saturate_cast<unsigned>(buf[i]);
}
#else
modules/core/src/convert.simd.hpp
+1 -1
View File
@@ -432,7 +432,7 @@ DEF_CVT_FUNC(64f32u, cvt_64f, double, unsigned, v_float32)
DEF_CVT_FUNC(64f32s, cvt_, double, int, v_int32)
DEF_CVT_FUNC(64f32f, cvt_, double, float, v_float32)
DEF_CVT_FUNC(64f64u, cvt_64f, double, uint64_t, v_float64)
DEF_CVT_FUNC(64f64s, cvt_64f, double, int64_t, v_float32)
DEF_CVT_FUNC(64f64s, cvt_64f, double, int64_t, v_float64)
DEF_CVT_FUNC(64f16f, cvt1_,double, hfloat, v_float32)
DEF_CVT_FUNC(64f16bf, cvt1_,double, bfloat, v_float32)
DEF_CVT2BOOL_FUNC(64f8b, int64_t, 1)
modules/core/src/math.dispatch.cpp
+107
View File
@@ -0,0 +1,107 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// Dispatch layer for the element-wise MATH + SELECT kernels (math.simd.hpp) - the sibling of
// arithm.dispatch.cpp: plain functions forwarding to the CPU-optimal kernel via CV_CPU_DISPATCH.
// getElemwiseFunc (arithm.dispatch.cpp) routes the corresponding TOps here.
#include "precomp.hpp"
#include "arithm_expr.hpp"
#include "hal_replacement.hpp"
#include "math.simd.hpp"
#include "math.simd_declarations.hpp"
namespace cv { namespace ew {
// ---- pluggable-HAL bridge for exp/log --------------------------------------------------------
// The raw cv_hal_* entry points return int for a reason: without an installed HAL they are stubs
// returning CV_HAL_ERROR_NOT_IMPLEMENTED. getMathFunc PROBES each one once (a 1-element call on
// the safe input 1.0 - fine for both exp and log): implemented -> wrap it as an engine kernel
// (the function pointer rides in TKernel::userdata, castKernel-style) and the engine adds tiling
// and parallelism on top of the vendor code; not implemented -> the engine's own v_exp/v_log
// kernels. Uniform over ANY HAL (an external vendor one, the IPP HAL module, ...) - the get is
// called once per program build, the probe cost is nothing next to the kernel calls that follow.
template<typename T>
static int halUnaryKernel(const void* src0_, size_t s0y, size_t s0x,
const void*, size_t, size_t, const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int, void* userdata)
{
typedef int (*HalFunc)(const T*, T*, int);
const HalFunc fn = (HalFunc)userdata;
s0y /= sizeof(T);
dsty /= sizeof(T);
CV_Assert(s0x <= 1u);
const T* src0 = (const T*)src0_;
T* dst = (T*)dst_;
if (height > 1 && dsty == (size_t)width && s0y == s0x*(size_t)width) { width *= height; height = 1; }
const int urows = (s0y == 0 && height > 1) ? 1 : height; // vertical broadcast: 1 row + copies
for (int y = 0; y < urows; y++, src0 += s0y, dst += dsty)
{
int code;
if (s0x == 0) // broadcast-scalar source: one value covers the row
{
T v;
code = fn(src0, &v, 1);
for (int x = 0; x < width; x++) dst[x] = v;
}
else
code = fn(src0, dst, width);
if (code != CV_HAL_ERROR_OK)
return code; // shouldn't happen (probed at get time) - let exec assert
}
dst = (T*)dst_;
for (int y = urows; y < height; y++)
memcpy(dst + (size_t)y*dsty, dst, (size_t)width*sizeof(T));
return 0;
}
template<typename T, typename HalFunc>
static TKernel probeHalUnary(HalFunc fn)
{
T one = (T)1, r = (T)0;
if (fn(&one, &r, 1) == CV_HAL_ERROR_OK)
return {halUnaryKernel<T>, (void*)fn, 0};
return {};
}
// The engine's OWN kernel for (op, T) - v_exp/v_log & co, no HAL/IPP tiers. The final fallback of
// getMathFunc, and what hal::exp32f & co use as THEIR built-in implementation (the former table
// kernels are gone), via mathSpanEngine below.
static TKernel getEngineMathFunc(TOp op, int T)
{
CV_CPU_DISPATCH(getMathFunc_, (op, T), CV_CPU_DISPATCH_MODES_ALL);
}
// run the engine's own math kernel over one contiguous span (the shape hal::exp32f & co need)
void mathSpanEngine(TOp op, int depth, const void* src, void* dst, int n)
{
TKernel k = getEngineMathFunc(op, depth);
CV_Assert(k.fptr);
const double noparams[4] = {};
k.fptr(src, 0, 1, nullptr, 0, 0, nullptr, 0, 0, dst, 0, n, 1, noparams, k.flags, k.userdata);
}
TKernel getMathFunc(TOp op, int T)
{
if ((op == OP_EXP || op == OP_LOG) && (T == CV_32F || T == CV_64F))
{
// IPP now rides the cv_hal_* hooks too (hal/ipp), so the single probe below picks it up
// together with any external vendor HAL - no separate IPP tier needed here.
// probe results are process-lifetime stable; cache them (thread-safe magic statics)
static const TKernel exp32 = probeHalUnary<float >(cv_hal_exp32f);
static const TKernel exp64 = probeHalUnary<double>(cv_hal_exp64f);
static const TKernel log32 = probeHalUnary<float >(cv_hal_log32f);
static const TKernel log64 = probeHalUnary<double>(cv_hal_log64f);
const TKernel* k = op == OP_EXP ? (T == CV_32F ? &exp32 : &exp64)
: (T == CV_32F ? &log32 : &log64);
if (k->fptr)
return *k;
}
return getEngineMathFunc(op, T);
}
TKernel getPowFunc(int T, int R) { CV_CPU_DISPATCH(getPowFunc_, (T, R), CV_CPU_DISPATCH_MODES_ALL); }
}} // namespace cv::ew
modules/core/src/math.simd.hpp
+487
View File
@@ -0,0 +1,487 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// Element-wise MATH kernels (sqrt/exp/log/sin/cos/tanh/erf/relu) and SELECT for the new arithmetic
// engine, SIMD-dispatched per CPU baseline - the unary/ternary sibling of arithm.simd.hpp.
//
// This file is compiled once per SIMD baseline (registered via ocv_add_dispatched_file). The per-op
// entry points get*Func_(...) live in cv::ew::CV_CPU_OPTIMIZATION_NAMESPACE and return the kernel
// optimized for that baseline; the regular get*Func dispatchers live in math.dispatch.cpp.
//
// Kernel shape (house style of arithm.simd.hpp):
// - one 2D tile; per-row outer loop with stepy (bytes); dst contiguous in x; stepx in {0,1}.
// - continuity collapse 2D->1D when every operand+dst is gap-free.
// - halide right-edge backoff for the SIMD tail, SUPPRESSED when dst aliases an input (in-place
// unary math would re-apply Op to already-written values).
//
// Math is T -> T over the four float depths: f32/f64 compute natively (v_exp & co exist for both);
// f16/bf16 ride the f32 hub (vx_load_pair_as widens one native vector into two f32 vectors, the
// saturating v_store_pair_as packs them back) - more accurate than a native f16 polynomial and
// works on every baseline. Integer inputs never reach these kernels: emitUnary computes integer
// math in the float domain and casts.
#include "opencv2/core/hal/intrin.hpp"
#include "convert.hpp" // typed vx_load_pair_as / v_store_pair_as helpers (cv::)
#include "arithm_expr.hpp" // the kernel contract: TOp / TKernel / KernelFunc
#include <cmath>
namespace cv {
// Everything outside cv::ew::CV_CPU_OPTIMIZATION_NAMESPACE must be skipped in the
// declarations-only re-includes (one per dispatched mode), or it gets redefined.
#ifndef CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
#if (CV_SIMD || CV_SIMD_SCALABLE)
// f32-pair -> f16/bf16 stores for the half-float hub (convert.hpp covers the other pairs)
static inline void v_store_pair_as(hfloat* p, const v_float32& a, const v_float32& b)
{
v_pack_store(p, a);
v_pack_store(p + VTraits<v_float32>::vlanes(), b);
}
static inline void v_store_pair_as(bfloat* p, const v_float32& a, const v_float32& b)
{
v_pack_store(p, a);
v_pack_store(p + VTraits<v_float32>::vlanes(), b);
}
#endif
#endif // CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
namespace ew {
CV_CPU_OPTIMIZATION_NAMESPACE_BEGIN
// ---- per-op kernel entry points for THIS baseline (the regular dispatchers in
// math.dispatch.cpp reach them through CV_CPU_DISPATCH). ----
TKernel getMathFunc_(TOp op, int T); // unary math, T -> T, T in {f16, bf16, f32, f64}
TKernel getPowFunc_(int T, int R); // OP_POW, T x T -> T (R must equal T)
#ifndef CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
// ===========================================================================
// Op functors: vec(Wvec) over the work vector (f32 or f64), scl(WT) for the scalar path/tail.
// New unary math ops slot in here.
// ===========================================================================
struct MSqrt {
template<typename V> static V vec(const V& x) { return v_sqrt(x); }
template<typename W> static W scl(W x) { return std::sqrt(x); }
};
struct MExp {
template<typename V> static V vec(const V& x) { return v_exp(x); }
template<typename W> static W scl(W x) { return std::exp(x); }
};
struct MLog {
template<typename V> static V vec(const V& x) { return v_log(x); }
template<typename W> static W scl(W x) { return std::log(x); }
};
struct MSin {
template<typename V> static V vec(const V& x) { return v_sin(x); }
template<typename W> static W scl(W x) { return std::sin(x); }
};
struct MCos {
template<typename V> static V vec(const V& x) { return v_cos(x); }
template<typename W> static W scl(W x) { return std::cos(x); }
};
// tanh(x) = (e^2x - 1) / (e^2x + 1), on top of v_exp (no v_tanh intrinsic). The input is clamped
// first: tanh saturates to +/-1 well inside |x| <= 10 (f32) / 20 (f64), while an unclamped large x
// would push e^2x to inf and the ratio to inf/inf = NaN. (A NaN input may map to a saturated value
// on some ISAs instead of NaN - the polynomial v_exp has relaxed NaN semantics anyway.)
struct MTanh {
static v_float32 vec(const v_float32& x)
{
const v_float32 one = vx_setall_f32(1.f), lim = vx_setall_f32(10.f);
v_float32 cx = v_min(v_max(x, v_sub(vx_setzero_f32(), lim)), lim);
v_float32 e = v_exp(v_add(cx, cx));
return v_div(v_sub(e, one), v_add(e, one));
}
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
static v_float64 vec(const v_float64& x)
{
const v_float64 one = vx_setall_f64(1.), lim = vx_setall_f64(20.);
v_float64 cx = v_min(v_max(x, v_sub(vx_setzero_f64(), lim)), lim);
v_float64 e = v_exp(v_add(cx, cx));
return v_div(v_sub(e, one), v_add(e, one));
}
#endif
template<typename W> static W scl(W x) { return std::tanh(x); }
};
struct MErf {
static v_float32 vec(const v_float32& x) { return v_erf(x); }
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
// no f64 SIMD erf primitive: apply std::erf per lane (keeps the one kernel shape; the
// store/load round-trip is noise next to libm erf itself)
static v_float64 vec(const v_float64& x)
{
double buf[VTraits<v_float64>::max_nlanes];
v_store(buf, x);
for (int i = 0; i < VTraits<v_float64>::vlanes(); i++) buf[i] = std::erf(buf[i]);
return vx_load(buf);
}
#endif
template<typename W> static W scl(W x) { return std::erf(x); }
};
struct MRelu {
template<typename V> static V vec(const V& x) { return v_max(x, v_setzero_<V>()); }
template<typename W> static W scl(W x) { return x > W(0) ? x : W(0); }
};
// ===========================================================================
// The unary kernel: dst = Op(src), T -> T. Wvec picks the work vector: v_float32 / v_float64 for
// the native depths, v_float32 for the f16/bf16 hub (vx_load_pair_as does the widening).
// ===========================================================================
template<typename T, typename Wvec, class Op>
static int vecUnaryKernel(const void* src0_, size_t s0y, size_t s0x,
const void*, size_t, size_t, const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s0y /= sizeof(T);
dsty /= sizeof(T);
CV_Assert(s0x <= 1u);
const T* src0 = (const T*)src0_;
T* dst = (T*)dst_;
using WT = typename VTraits<Wvec>::lane_type;
if (height > 1 && dsty == (size_t)width && s0y == s0x*(size_t)width) { width *= height; height = 1; }
// vertical broadcast (a row expanded into a matrix, s0y == 0): every output row is identical -
// compute the first one, memcpy the rest (transcendentals cost far more than a row copy)
const int urows = (s0y == 0 && height > 1) ? 1 : height;
for (int y = 0; y < urows; y++, src0 += s0y, dst += dsty)
{
if (s0x == 0) // broadcast-scalar source: one value covers the row
{
T v = saturate_cast<T>(Op::scl((WT)src0[0]));
for (int x = 0; x < width; x++) dst[x] = v;
continue;
}
int x = 0;
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int VECSZ = VTraits<Wvec>::vlanes();
// in-place (dst == src) forbids the right-edge backoff: it would re-read already-written
// values and apply Op twice. Those rows finish in the scalar tail instead.
const bool use_tail_trick = width >= VECSZ*4 && src0_ != dst_;
for (; x < width; x += VECSZ*2)
{
if (x + VECSZ*2 > width) { if (!use_tail_trick || x == 0) break; x = width - VECSZ*2; }
Wvec a0, a1;
vx_load_pair_as(src0 + x, a0, a1);
a0 = Op::vec(a0); a1 = Op::vec(a1);
v_store_pair_as(dst + x, a0, a1);
}
#endif
for (; x < width; x++)
dst[x] = saturate_cast<T>(Op::scl((WT)src0[x]));
}
dst = (T*)dst_;
for (int y = urows; y < height; y++)
memcpy(dst + (size_t)y*dsty, dst, (size_t)width*sizeof(T));
return 0;
}
// Scalar-only variant for (op, depth) pairs with no SIMD primitive (erf on f64; every op's f64
// when the baseline has no 64-bit float SIMD).
template<typename T, typename WT, class Op>
static int scalarUnaryKernel(const void* src0_, size_t s0y, size_t s0x,
const void*, size_t, size_t, const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s0y /= sizeof(T);
dsty /= sizeof(T);
CV_Assert(s0x <= 1u);
const T* src0 = (const T*)src0_;
T* dst = (T*)dst_;
if (height > 1 && dsty == (size_t)width && s0y == s0x*(size_t)width) { width *= height; height = 1; }
const int urows = (s0y == 0 && height > 1) ? 1 : height; // vertical broadcast: 1 row + copies
for (int y = 0; y < urows; y++, src0 += s0y, dst += dsty)
{
if (s0x == 0)
{
T v = saturate_cast<T>(Op::scl((WT)src0[0]));
for (int x = 0; x < width; x++) dst[x] = v;
continue;
}
for (int x = 0; x < width; x++)
dst[x] = saturate_cast<T>(Op::scl((WT)src0[x]));
}
dst = (T*)dst_;
for (int y = urows; y < height; y++)
memcpy(dst + (size_t)y*dsty, dst, (size_t)width*sizeof(T));
return 0;
}
// ===========================================================================
// OP_POW: dst = pow(x, y), T x T -> T over the float depths. Exact std::pow semantics.
//
// The exponent is USUALLY a broadcast scalar (pow(x, 2), texpr literals ride as 0-dim consts with
// stepx == 0) - dispatched PER ROW to the important special cases: y==2 -> x*x, y==3 -> x*x*x,
// y==0.5 -> v_sqrt, y==1 -> copy, y==0 -> fill 1 (std::pow(anything, 0) == 1, NaN included).
// Everything else - and the per-element exponent - runs the general vectorized path
// exp(y * log(x)), which is only valid for x > 0: any lane with x <= 0 falls back to scalar
// std::pow for the whole vector pair (v_check_any per pair; negative/zero bases are rare, and the
// scalar path preserves every std::pow subtlety - signed results for integer y, NaN for
// fractional y, the x == 0 family). One knowing deviation: y==0.5 uses v_sqrt, so pow(-0., .5)
// returns -0. instead of std::pow's +0.
//
// The halide right-edge tail backoff is used in every SIMD loop, SUPPRESSED when dst aliases an
// input: pow is not idempotent, so an in-place backoff would re-read already-written values (the
// overlap region is otherwise just recomputed from the untouched source). Suppressed rows finish
// in the scalar tail.
#if (CV_SIMD || CV_SIMD_SCALABLE)
static inline v_float32 vxSetallW(float v, const v_float32&) { return vx_setall_f32(v); }
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
static inline v_float64 vxSetallW(double v, const v_float64&) { return vx_setall_f64(v); }
#endif
#endif
// Plain scalar pow for baselines without the needed SIMD float width (f64 without 64-bit SIMD).
template<typename T, typename WT>
static int scalarPowKernel(const void* src0_, size_t s0y, size_t s0x,
const void* src1_, size_t s1y, size_t s1x,
const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s0y /= sizeof(T); s1y /= sizeof(T); dsty /= sizeof(T);
CV_Assert(s0x <= 1u && s1x <= 1u);
const T* src0 = (const T*)src0_;
const T* src1 = (const T*)src1_;
T* dst = (T*)dst_;
if (height > 1 && dsty == (size_t)width && s0y == s0x*(size_t)width && s1y == s1x*(size_t)width)
{ width *= height; height = 1; }
for (int y = 0; y < height; y++, src0 += s0y, src1 += s1y, dst += dsty)
for (int x = 0; x < width; x++)
dst[x] = saturate_cast<T>(std::pow((WT)src0[x*s0x], (WT)src1[x*s1x]));
return 0;
}
template<typename T, typename Wvec>
static int powKernel(const void* src0_, size_t s0y, size_t s0x,
const void* src1_, size_t s1y, size_t s1x,
const void*, size_t, size_t,
void* dst_, size_t dsty, int width, int height,
const double*, int, void*)
{
s0y /= sizeof(T);
s1y /= sizeof(T);
dsty /= sizeof(T);
CV_Assert(s0x <= 1u && s1x <= 1u);
const T* src0 = (const T*)src0_;
const T* src1 = (const T*)src1_;
T* dst = (T*)dst_;
using WT = typename VTraits<Wvec>::lane_type;
if (height > 1 && dsty == (size_t)width && s0y == s0x*(size_t)width && s1y == s1x*(size_t)width)
{ width *= height; height = 1; }
[[maybe_unused]] const bool tail_trick = src0_ != dst_ && src1_ != dst_;
// both operands vertically broadcast: every output row is identical - compute one, copy
const int urows = (s0y == 0 && s1y == 0 && height > 1) ? 1 : height;
for (int y = 0; y < urows; y++, src0 += s0y, src1 += s1y, dst += dsty)
{
int x = 0;
if (s1x == 0) // scalar exponent for this row
{
const WT p = (WT)src1[0];
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int VECSZ = VTraits<Wvec>::vlanes();
if (s0x == 1)
{
if (p == WT(2) || p == WT(3))
{
for (; x < width; x += VECSZ*2)
{
if (x + VECSZ*2 > width) { if (!tail_trick || x == 0) break; x = width - VECSZ*2; }
Wvec a0, a1;
vx_load_pair_as(src0 + x, a0, a1);
Wvec r0 = v_mul(a0, a0), r1 = v_mul(a1, a1);
if (p == WT(3)) { r0 = v_mul(r0, a0); r1 = v_mul(r1, a1); }
v_store_pair_as(dst + x, r0, r1);
}
}
else if (p == WT(0.5))
{
for (; x < width; x += VECSZ*2)
{
if (x + VECSZ*2 > width) { if (!tail_trick || x == 0) break; x = width - VECSZ*2; }
Wvec a0, a1;
vx_load_pair_as(src0 + x, a0, a1);
a0 = v_sqrt(a0); a1 = v_sqrt(a1);
v_store_pair_as(dst + x, a0, a1);
}
}
else if (p == WT(1))
{
if ((const void*)src0 != (const void*)dst)
for (; x < width; x++) dst[x] = src0[x];
x = width;
}
else if (p == WT(0))
{
const T one = saturate_cast<T>(1);
for (; x < width; x++) dst[x] = one;
}
else if (p == WT(-0.5))
{
const Wvec one = vxSetallW(WT(1), Wvec());
for (; x < width; x += VECSZ*2)
{
if (x + VECSZ*2 > width) { if (!tail_trick || x == 0) break; x = width - VECSZ*2; }
Wvec a0, a1;
vx_load_pair_as(src0 + x, a0, a1);
a0 = v_div(one, v_sqrt(a0)); a1 = v_div(one, v_sqrt(a1));
v_store_pair_as(dst + x, a0, a1);
}
}
else if (p == std::rint(p) && std::abs(p) <= WT(65536))
{
// any other INTEGER exponent: LSB-first binary exponentiation - the same
// multiply chain (and order) as the classic iPow, fully vectorized. Also more
// accurate than exp(p*log x) (a few ulp vs ~2e-7 rel) and semantically exact
// on non-positive bases: the sign falls out of the multiplies, 0^negative
// divides to inf - no scalar patching needed.
const int ip = (int)p, ap = ip < 0 ? -ip : ip; // ap >= 1 (0..3 handled above)
const Wvec one = vxSetallW(WT(1), Wvec());
for (; x < width; x += VECSZ*2)
{
if (x + VECSZ*2 > width) { if (!tail_trick || x == 0) break; x = width - VECSZ*2; }
Wvec b0, b1;
vx_load_pair_as(src0 + x, b0, b1);
Wvec a0 = one, a1 = one;
for (int q = ap; q > 1; q >>= 1)
{
if (q & 1) { a0 = v_mul(a0, b0); a1 = v_mul(a1, b1); }
b0 = v_mul(b0, b0); b1 = v_mul(b1, b1);
}
a0 = v_mul(a0, b0); a1 = v_mul(a1, b1);
if (ip < 0) { a0 = v_div(one, a0); a1 = v_div(one, a1); }
v_store_pair_as(dst + x, a0, a1);
}
}
else // general scalar exponent: exp(p * log(x))
{
const Wvec vp = vxSetallW(p, Wvec()), z = v_setzero_<Wvec>();
for (; x < width; x += VECSZ*2)
{
if (x + VECSZ*2 > width) { if (!tail_trick || x == 0) break; x = width - VECSZ*2; }
Wvec a0, a1;
vx_load_pair_as(src0 + x, a0, a1);
if (v_check_any(v_le(a0, z)) || v_check_any(v_le(a1, z)))
{ // exact std::pow for x <= 0 lanes
for (int i = 0; i < VECSZ*2; i++)
dst[x + i] = saturate_cast<T>(std::pow((WT)src0[x + i], p));
continue;
}
a0 = v_exp(v_mul(vp, v_log(a0)));
a1 = v_exp(v_mul(vp, v_log(a1)));
v_store_pair_as(dst + x, a0, a1);
}
}
}
#endif
for (; x < width; x++)
dst[x] = saturate_cast<T>(std::pow((WT)src0[x*s0x], p));
continue;
}
// per-element exponent
#if (CV_SIMD || CV_SIMD_SCALABLE)
if (s0x == 1)
{
const int VECSZ = VTraits<Wvec>::vlanes();
const Wvec z = v_setzero_<Wvec>();
for (; x < width; x += VECSZ*2)
{
if (x + VECSZ*2 > width) { if (!tail_trick || x == 0) break; x = width - VECSZ*2; }
Wvec a0, a1, b0, b1;
vx_load_pair_as(src0 + x, a0, a1);
vx_load_pair_as(src1 + x, b0, b1);
if (v_check_any(v_le(a0, z)) || v_check_any(v_le(a1, z)))
{
for (int i = 0; i < VECSZ*2; i++)
dst[x + i] = saturate_cast<T>(std::pow((WT)src0[x + i], (WT)src1[x + i]));
continue;
}
a0 = v_exp(v_mul(b0, v_log(a0)));
a1 = v_exp(v_mul(b1, v_log(a1)));
v_store_pair_as(dst + x, a0, a1);
}
}
#endif
for (; x < width; x++)
dst[x] = saturate_cast<T>(std::pow((WT)src0[x*s0x], (WT)src1[x]));
}
dst = (T*)dst_;
for (int y = urows; y < height; y++)
memcpy(dst + (size_t)y*dsty, dst, (size_t)width*sizeof(T));
return 0;
}
TKernel getPowFunc_(int T, int R)
{
if (R != T)
return {};
KernelFunc fptr = nullptr;
switch (T)
{
case CV_16F: fptr = powKernel<hfloat, v_float32>; break;
case CV_16BF: fptr = powKernel<bfloat, v_float32>; break;
case CV_32F: fptr = powKernel<float, v_float32>; break;
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
case CV_64F: fptr = powKernel<double, v_float64>; break;
#else
case CV_64F: fptr = scalarPowKernel<double, double>; break;
#endif
default: ;
}
return {fptr, nullptr, 0};
}
// ===========================================================================
// getters for THIS baseline
// ===========================================================================
template<class Op>
static KernelFunc mathByDepth(int T)
{
switch (T)
{
case CV_16F: return vecUnaryKernel<hfloat, v_float32, Op>;
case CV_16BF: return vecUnaryKernel<bfloat, v_float32, Op>;
case CV_32F: return vecUnaryKernel<float, v_float32, Op>;
#if CV_SIMD_64F || CV_SIMD_SCALABLE_64F
case CV_64F: return vecUnaryKernel<double, v_float64, Op>;
#else
case CV_64F: return scalarUnaryKernel<double, double, Op>;
#endif
default: return nullptr;
}
}
TKernel getMathFunc_(TOp op, int T)
{
KernelFunc f = nullptr;
switch (op)
{
case OP_SQRT: f = mathByDepth<MSqrt>(T); break;
case OP_EXP: f = mathByDepth<MExp >(T); break;
case OP_LOG: f = mathByDepth<MLog >(T); break;
case OP_SIN: f = mathByDepth<MSin >(T); break;
case OP_COS: f = mathByDepth<MCos >(T); break;
case OP_TANH: f = mathByDepth<MTanh>(T); break;
case OP_RELU: f = mathByDepth<MRelu>(T); break;
case OP_ERF: f = mathByDepth<MErf >(T); break;
default: ;
}
return {f, nullptr, 0};
}
#endif // CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
CV_CPU_OPTIMIZATION_NAMESPACE_END
}} // namespace cv::ew
modules/core/src/mathfuncs.cpp
+104 -522
View File
@@ -49,6 +49,7 @@
#include <algorithm>
#include <cmath>
#include "mathfuncs.hpp"
#include "arithm_expr.hpp" // the element-wise engine: getMathFunc + TExpr for cv::exp/log/sqrt
namespace cv
{
@@ -436,32 +437,57 @@ void polarToCart( InputArray src1, InputArray src2,
* E X P *
\****************************************************************************************/
// The master function of the unary math family (cv::exp/log/sqrt - the analogue of arithm_op):
// same-shape same-type output over the four float depths, computed by the element-wise engine's
// kernels (math.simd.hpp). Two tiers:
// - SMALL and continuous (a common pattern - exp() over one image row as a lookup substitute):
// call the kernel DIRECTLY over the flattened elements. No TExpr, no broadcastOp, no
// parallel_for machinery - their setup dominates at these sizes.
// - everything else (large arrays - worth parallelizing; ROIs - need real steps): the usual
// 1-instruction program via compile()/exec().
enum { MATH_OP_SMALL = 100000 }; // elements; tune with a benchmark if the crossover moves
static void math_op(ew::TOp op, InputArray _src, OutputArray _dst)
{
int type = _src.type(), depth = CV_MAT_DEPTH(type);
CV_Assert(depth == CV_16F || depth == CV_16BF || depth == CV_32F || depth == CV_64F);
Mat src = _src.getMat();
_dst.createSameSize(_src, type); // whole-shape transfer (layout & future metadata included)
Mat dst = _dst.getMat();
if (src.empty())
return;
const size_t total = src.total() * src.channels();
if (src.isContinuous() && dst.isContinuous() && total <= (size_t)MATH_OP_SMALL)
{
ew::TKernel k = ew::getMathFunc(op, depth);
CV_Assert(k.fptr);
static const double noparams[4] = {};
k.fptr(src.data, 0, 1, nullptr, 0, 0, nullptr, 0, 0,
dst.data, 0, (int)total, 1, noparams, k.flags, k.userdata);
return;
}
ew::TExpr p;
const int a = p.addInput(depth);
const int out = p.addOutput(depth);
p.moveToOutput(p.emitUnary(op, a, depth), out);
p.compile();
const Mat* inputs[] = { &src };
p.exec(inputs, &dst);
}
void exp( InputArray _src, OutputArray _dst )
{
CV_INSTRUMENT_REGION();
int type = _src.type(), depth = _src.depth(), cn = _src.channels();
CV_Assert( depth == CV_32F || depth == CV_64F );
[[maybe_unused]] int depth = _src.depth(); // consumed by CV_OCL_RUN only
CV_OCL_RUN(_dst.isUMat() && _src.dims() <= 2,
CV_OCL_RUN(_dst.isUMat() && _src.dims() <= 2 && (depth == CV_32F || depth == CV_64F),
ocl_math_op(_src, noArray(), _dst, OCL_OP_EXP))
Mat src = _src.getMat();
_dst.create( src.size, type );
Mat dst = _dst.getMat();
const Mat* arrays[] = {&src, &dst, 0};
uchar* ptrs[2] = {};
NAryMatIterator it(arrays, ptrs);
int len = (int)(it.size*cn);
for( size_t i = 0; i < it.nplanes; i++, ++it )
{
if( depth == CV_32F )
hal::exp32f((const float*)ptrs[0], (float*)ptrs[1], len);
else
hal::exp64f((const double*)ptrs[0], (double*)ptrs[1], len);
}
math_op(ew::OP_EXP, _src, _dst);
}
@@ -473,28 +499,12 @@ void log( InputArray _src, OutputArray _dst )
{
CV_INSTRUMENT_REGION();
int type = _src.type(), depth = _src.depth(), cn = _src.channels();
CV_Assert( depth == CV_32F || depth == CV_64F );
[[maybe_unused]] int depth = _src.depth(); // consumed by CV_OCL_RUN only
CV_OCL_RUN( _dst.isUMat() && _src.dims() <= 2,
CV_OCL_RUN( _dst.isUMat() && _src.dims() <= 2 && (depth == CV_32F || depth == CV_64F),
ocl_math_op(_src, noArray(), _dst, OCL_OP_LOG))
Mat src = _src.getMat();
_dst.create( src.size, type );
Mat dst = _dst.getMat();
const Mat* arrays[] = {&src, &dst, 0};
uchar* ptrs[2] = {};
NAryMatIterator it(arrays, ptrs);
int len = (int)(it.size*cn);
for( size_t i = 0; i < it.nplanes; i++, ++it )
{
if( depth == CV_32F )
hal::log32f( (const float*)ptrs[0], (float*)ptrs[1], len );
else
hal::log64f( (const double*)ptrs[0], (double*)ptrs[1], len );
}
math_op(ew::OP_LOG, _src, _dst);
}
/****************************************************************************************\
@@ -1033,133 +1043,81 @@ void pow( InputArray _src, double power, OutputArray _dst )
CV_OCL_RUN(useOpenCL, ocl_pow(_src, power, _dst, is_ipower, ipower))
Mat src = _src.getMat();
_dst.create( src.size, type );
Mat dst = _dst.getMat();
const bool floatDepth = depth == CV_16F || depth == CV_16BF || depth == CV_32F || depth == CV_64F;
const Mat* arrays[] = {&src, &dst, 0};
uchar* ptrs[2] = {};
NAryMatIterator it(arrays, ptrs);
int len = (int)(it.size*cn);
if( is_ipower )
// INTEGER array ** INTEGER power: keep the classic iPow kernels (an exact multiply chain with
// the classic wrap-around semantics) - full bit-exact compatibility for whoever relies on it.
// Everything else - any power on a float array, a fractional power on an integer one (computed
// in the float domain and saturated back), plus the 32U/64-bit depths iPow never supported -
// goes through the engine below.
if( is_ipower && !floatDepth && ipowTab[depth] )
{
Mat src = _src.getMat();
_dst.createSameSize(_src, type);
Mat dst = _dst.getMat();
const Mat* arrays[] = {&src, &dst, 0};
uchar* ptrs[2] = {};
NAryMatIterator it(arrays, ptrs);
int len = (int)(it.size*cn);
IPowFunc func = ipowTab[depth];
CV_Assert( func != 0 );
for( size_t i = 0; i < it.nplanes; i++, ++it )
func( ptrs[0], ptrs[1], len, ipower );
return;
}
else if( fabs(fabs(power) - 0.5) < DBL_EPSILON )
// The engine path, two tiers like math_op: the pow kernel special-cases the exponents
// 3/0.5 (2/1/0 never reach here) per row and vectorizes the general exp(p*log x) with an exact
// std::pow patch for non-positive bases (0^negative -> inf, negative^fractional -> NaN).
Mat src = _src.getMat();
_dst.createSameSize(_src, type);
Mat dst = _dst.getMat();
if (src.empty())
return;
const size_t total = src.total() * cn;
if (floatDepth && src.isContinuous() && dst.isContinuous() && total <= (size_t)MATH_OP_SMALL)
{
MathFunc func = power < 0 ?
(depth == CV_32F ? (MathFunc)hal::invSqrt32f : (MathFunc)hal::invSqrt64f) :
(depth == CV_32F ? (MathFunc)hal::sqrt32f : (MathFunc)hal::sqrt64f);
for( size_t i = 0; i < it.nplanes; i++, ++it )
func( ptrs[0], ptrs[1], len );
}
else
{
int j, k, blockSize = std::min(len, ((BLOCK_SIZE + cn-1)/cn)*cn);
size_t esz1 = src.elemSize1();
AutoBuffer<uchar> buf;
Cv32suf inf32, nan32;
Cv64suf inf64, nan64;
float* fbuf = 0;
double* dbuf = 0;
#ifndef __EMSCRIPTEN__
inf32.i = 0x7f800000;
nan32.i = 0x7fffffff;
inf64.i = CV_BIG_INT(0x7FF0000000000000);
nan64.i = CV_BIG_INT(0x7FFFFFFFFFFFFFFF);
#else
inf32.f = std::numeric_limits<float>::infinity();
nan32.f = std::numeric_limits<float>::quiet_NaN();
inf64.f = std::numeric_limits<double>::infinity();
nan64.f = std::numeric_limits<double>::quiet_NaN();
#endif
if( src.ptr() == dst.ptr() )
ew::TKernel k = ew::getPowFunc(depth, depth);
if (k.fptr)
{
buf.allocate(blockSize*esz1);
fbuf = (float*)buf.data();
dbuf = (double*)buf.data();
}
for( size_t i = 0; i < it.nplanes; i++, ++it )
{
for( j = 0; j < len; j += blockSize )
double pvstore; // the broadcast exponent, stored as T
void* pv = &pvstore;
switch (depth)
{
int bsz = std::min(len - j, blockSize);
if( depth == CV_32F )
{
float* x0 = (float*)ptrs[0];
float* x = fbuf ? fbuf : x0;
float* y = (float*)ptrs[1];
if( x != x0 )
memcpy(x, x0, bsz*esz1);
hal::log32f(x, y, bsz);
for( k = 0; k < bsz; k++ )
y[k] = (float)(y[k]*power);
hal::exp32f(y, y, bsz);
for( k = 0; k < bsz; k++ )
{
if( x0[k] <= 0 )
{
if( x0[k] == 0.f )
{
if( power < 0 )
y[k] = inf32.f;
}
else
y[k] = nan32.f;
}
}
}
else
{
double* x0 = (double*)ptrs[0];
double* x = dbuf ? dbuf : x0;
double* y = (double*)ptrs[1];
if( x != x0 )
memcpy(x, x0, bsz*esz1);
hal::log64f(x, y, bsz);
for( k = 0; k < bsz; k++ )
y[k] *= power;
hal::exp64f(y, y, bsz);
for( k = 0; k < bsz; k++ )
{
if( x0[k] <= 0 )
{
if( x0[k] == 0. )
{
if( power < 0 )
y[k] = inf64.f;
}
else
y[k] = nan64.f;
}
}
}
ptrs[0] += bsz*esz1;
ptrs[1] += bsz*esz1;
case CV_16F: *(hfloat*)pv = saturate_cast<hfloat>(power); break;
case CV_16BF: *(bfloat*)pv = saturate_cast<bfloat>(power); break;
case CV_32F: *(float*)pv = (float)power; break;
default: pvstore = power; break;
}
static const double noparams[4] = {};
k.fptr(src.data, 0, 1, pv, 0, 0, nullptr, 0, 0,
dst.data, 0, (int)total, 1, noparams, k.flags, k.userdata);
return;
}
}
ew::TExpr prog;
const int a = prog.addInput(depth);
const int c = prog.addConst(ew::EW_DEPTH_NONE, Scalar(power), 1);
const int out = prog.addOutput(depth);
prog.moveToOutput(prog.emitBinary(ew::OP_POW, a, c, depth), out);
prog.compile();
const Mat* inputs[] = { &src };
prog.exec(inputs, &dst);
}
void sqrt(InputArray a, OutputArray b)
{
CV_INSTRUMENT_REGION();
cv::pow(a, 0.5, b);
if (b.isUMat() && a.dims() <= 2) // the OpenCL route (via ocl_pow) is unchanged
{
cv::pow(a, 0.5, b);
return;
}
math_op(ew::OP_SQRT, a, b);
}
/************************** CheckArray for NaN's, Inf's *********************************/
@@ -1702,381 +1660,5 @@ double cv::solvePoly( InputArray _coeffs0, OutputArray _roots0, int maxIters )
return maxDiff;
}
// Common constants for dispatched code
namespace cv { namespace details {
#define EXPTAB_SCALE 6
#define EXPTAB_MASK ((1 << EXPTAB_SCALE) - 1)
#define EXPPOLY_32F_A0 .9670371139572337719125840413672004409288e-2
static const double CV_DECL_ALIGNED(64) expTab[EXPTAB_MASK + 1] = {
1.0 * EXPPOLY_32F_A0,
1.0108892860517004600204097905619 * EXPPOLY_32F_A0,
1.0218971486541166782344801347833 * EXPPOLY_32F_A0,
1.0330248790212284225001082839705 * EXPPOLY_32F_A0,
1.0442737824274138403219664787399 * EXPPOLY_32F_A0,
1.0556451783605571588083413251529 * EXPPOLY_32F_A0,
1.0671404006768236181695211209928 * EXPPOLY_32F_A0,
1.0787607977571197937406800374385 * EXPPOLY_32F_A0,
1.0905077326652576592070106557607 * EXPPOLY_32F_A0,
1.1023825833078409435564142094256 * EXPPOLY_32F_A0,
1.1143867425958925363088129569196 * EXPPOLY_32F_A0,
1.126521618608241899794798643787 * EXPPOLY_32F_A0,
1.1387886347566916537038302838415 * EXPPOLY_32F_A0,
1.151189229952982705817759635202 * EXPPOLY_32F_A0,
1.1637248587775775138135735990922 * EXPPOLY_32F_A0,
1.1763969916502812762846457284838 * EXPPOLY_32F_A0,
1.1892071150027210667174999705605 * EXPPOLY_32F_A0,
1.2021567314527031420963969574978 * EXPPOLY_32F_A0,
1.2152473599804688781165202513388 * EXPPOLY_32F_A0,
1.2284805361068700056940089577928 * EXPPOLY_32F_A0,
1.2418578120734840485936774687266 * EXPPOLY_32F_A0,
1.2553807570246910895793906574423 * EXPPOLY_32F_A0,
1.2690509571917332225544190810323 * EXPPOLY_32F_A0,
1.2828700160787782807266697810215 * EXPPOLY_32F_A0,
1.2968395546510096659337541177925 * EXPPOLY_32F_A0,
1.3109612115247643419229917863308 * EXPPOLY_32F_A0,
1.3252366431597412946295370954987 * EXPPOLY_32F_A0,
1.3396675240533030053600306697244 * EXPPOLY_32F_A0,
1.3542555469368927282980147401407 * EXPPOLY_32F_A0,
1.3690024229745906119296011329822 * EXPPOLY_32F_A0,
1.3839098819638319548726595272652 * EXPPOLY_32F_A0,
1.3989796725383111402095281367152 * EXPPOLY_32F_A0,
1.4142135623730950488016887242097 * EXPPOLY_32F_A0,
1.4296133383919700112350657782751 * EXPPOLY_32F_A0,
1.4451808069770466200370062414717 * EXPPOLY_32F_A0,
1.4609177941806469886513028903106 * EXPPOLY_32F_A0,
1.476826145939499311386907480374 * EXPPOLY_32F_A0,
1.4929077282912648492006435314867 * EXPPOLY_32F_A0,
1.5091644275934227397660195510332 * EXPPOLY_32F_A0,
1.5255981507445383068512536895169 * EXPPOLY_32F_A0,
1.5422108254079408236122918620907 * EXPPOLY_32F_A0,
1.5590044002378369670337280894749 * EXPPOLY_32F_A0,
1.5759808451078864864552701601819 * EXPPOLY_32F_A0,
1.5931421513422668979372486431191 * EXPPOLY_32F_A0,
1.6104903319492543081795206673574 * EXPPOLY_32F_A0,
1.628027421857347766848218522014 * EXPPOLY_32F_A0,
1.6457554781539648445187567247258 * EXPPOLY_32F_A0,
1.6636765803267364350463364569764 * EXPPOLY_32F_A0,
1.6817928305074290860622509524664 * EXPPOLY_32F_A0,
1.7001063537185234695013625734975 * EXPPOLY_32F_A0,
1.7186192981224779156293443764563 * EXPPOLY_32F_A0,
1.7373338352737062489942020818722 * EXPPOLY_32F_A0,
1.7562521603732994831121606193753 * EXPPOLY_32F_A0,
1.7753764925265212525505592001993 * EXPPOLY_32F_A0,
1.7947090750031071864277032421278 * EXPPOLY_32F_A0,
1.8142521755003987562498346003623 * EXPPOLY_32F_A0,
1.8340080864093424634870831895883 * EXPPOLY_32F_A0,
1.8539791250833855683924530703377 * EXPPOLY_32F_A0,
1.8741676341102999013299989499544 * EXPPOLY_32F_A0,
1.8945759815869656413402186534269 * EXPPOLY_32F_A0,
1.9152065613971472938726112702958 * EXPPOLY_32F_A0,
1.9360617934922944505980559045667 * EXPPOLY_32F_A0,
1.9571441241754002690183222516269 * EXPPOLY_32F_A0,
1.9784560263879509682582499181312 * EXPPOLY_32F_A0,
};
const double* getExpTab64f()
{
return expTab;
}
const float* getExpTab32f()
{
static float CV_DECL_ALIGNED(64) expTab_f[EXPTAB_MASK+1];
static std::atomic<bool> expTab_f_initialized(false);
if (!expTab_f_initialized.load())
{
for( int j = 0; j <= EXPTAB_MASK; j++ )
expTab_f[j] = (float)expTab[j];
expTab_f_initialized = true;
}
return expTab_f;
}
#define LOGTAB_SCALE 8
#define LOGTAB_MASK ((1 << LOGTAB_SCALE) - 1)
static const double CV_DECL_ALIGNED(64) logTab[(LOGTAB_MASK+1)*2] = {
0.0000000000000000000000000000000000000000, 1.000000000000000000000000000000000000000,
.00389864041565732288852075271279318258166, .9961089494163424124513618677042801556420,
.00778214044205494809292034119607706088573, .9922480620155038759689922480620155038760,
.01165061721997527263705585198749759001657, .9884169884169884169884169884169884169884,
.01550418653596525274396267235488267033361, .9846153846153846153846153846153846153846,
.01934296284313093139406447562578250654042, .9808429118773946360153256704980842911877,
.02316705928153437593630670221500622574241, .9770992366412213740458015267175572519084,
.02697658769820207233514075539915211265906, .9733840304182509505703422053231939163498,
.03077165866675368732785500469617545604706, .9696969696969696969696969696969696969697,
.03455238150665972812758397481047722976656, .9660377358490566037735849056603773584906,
.03831886430213659461285757856785494368522, .9624060150375939849624060150375939849624,
.04207121392068705056921373852674150839447, .9588014981273408239700374531835205992509,
.04580953603129420126371940114040626212953, .9552238805970149253731343283582089552239,
.04953393512227662748292900118940451648088, .9516728624535315985130111524163568773234,
.05324451451881227759255210685296333394944, .9481481481481481481481481481481481481481,
.05694137640013842427411105973078520037234, .9446494464944649446494464944649446494465,
.06062462181643483993820353816772694699466, .9411764705882352941176470588235294117647,
.06429435070539725460836422143984236754475, .9377289377289377289377289377289377289377,
.06795066190850773679699159401934593915938, .9343065693430656934306569343065693430657,
.07159365318700880442825962290953611955044, .9309090909090909090909090909090909090909,
.07522342123758751775142172846244648098944, .9275362318840579710144927536231884057971,
.07884006170777602129362549021607264876369, .9241877256317689530685920577617328519856,
.08244366921107458556772229485432035289706, .9208633093525179856115107913669064748201,
.08603433734180314373940490213499288074675, .9175627240143369175627240143369175627240,
.08961215868968712416897659522874164395031, .9142857142857142857142857142857142857143,
.09317722485418328259854092721070628613231, .9110320284697508896797153024911032028470,
.09672962645855109897752299730200320482256, .9078014184397163120567375886524822695035,
.10026945316367513738597949668474029749630, .9045936395759717314487632508833922261484,
.10379679368164355934833764649738441221420, .9014084507042253521126760563380281690141,
.10731173578908805021914218968959175981580, .8982456140350877192982456140350877192982,
.11081436634029011301105782649756292812530, .8951048951048951048951048951048951048951,
.11430477128005862852422325204315711744130, .8919860627177700348432055749128919860627,
.11778303565638344185817487641543266363440, .8888888888888888888888888888888888888889,
.12124924363286967987640707633545389398930, .8858131487889273356401384083044982698962,
.12470347850095722663787967121606925502420, .8827586206896551724137931034482758620690,
.12814582269193003360996385708858724683530, .8797250859106529209621993127147766323024,
.13157635778871926146571524895989568904040, .8767123287671232876712328767123287671233,
.13499516453750481925766280255629681050780, .8737201365187713310580204778156996587031,
.13840232285911913123754857224412262439730, .8707482993197278911564625850340136054422,
.14179791186025733629172407290752744302150, .8677966101694915254237288135593220338983,
.14518200984449788903951628071808954700830, .8648648648648648648648648648648648648649,
.14855469432313711530824207329715136438610, .8619528619528619528619528619528619528620,
.15191604202584196858794030049466527998450, .8590604026845637583892617449664429530201,
.15526612891112392955683674244937719777230, .8561872909698996655518394648829431438127,
.15860503017663857283636730244325008243330, .8533333333333333333333333333333333333333,
.16193282026931324346641360989451641216880, .8504983388704318936877076411960132890365,
.16524957289530714521497145597095368430010, .8476821192052980132450331125827814569536,
.16855536102980664403538924034364754334090, .8448844884488448844884488448844884488449,
.17185025692665920060697715143760433420540, .8421052631578947368421052631578947368421,
.17513433212784912385018287750426679849630, .8393442622950819672131147540983606557377,
.17840765747281828179637841458315961062910, .8366013071895424836601307189542483660131,
.18167030310763465639212199675966985523700, .8338762214983713355048859934853420195440,
.18492233849401198964024217730184318497780, .8311688311688311688311688311688311688312,
.18816383241818296356839823602058459073300, .8284789644012944983818770226537216828479,
.19139485299962943898322009772527962923050, .8258064516129032258064516129032258064516,
.19461546769967164038916962454095482826240, .8231511254019292604501607717041800643087,
.19782574332991986754137769821682013571260, .8205128205128205128205128205128205128205,
.20102574606059073203390141770796617493040, .8178913738019169329073482428115015974441,
.20421554142869088876999228432396193966280, .8152866242038216560509554140127388535032,
.20739519434607056602715147164417430758480, .8126984126984126984126984126984126984127,
.21056476910734961416338251183333341032260, .8101265822784810126582278481012658227848,
.21372432939771812687723695489694364368910, .8075709779179810725552050473186119873817,
.21687393830061435506806333251006435602900, .8050314465408805031446540880503144654088,
.22001365830528207823135744547471404075630, .8025078369905956112852664576802507836991,
.22314355131420973710199007200571941211830, .8000000000000000000000000000000000000000,
.22626367865045338145790765338460914790630, .7975077881619937694704049844236760124611,
.22937410106484582006380890106811420992010, .7950310559006211180124223602484472049689,
.23247487874309405442296849741978803649550, .7925696594427244582043343653250773993808,
.23556607131276688371634975283086532726890, .7901234567901234567901234567901234567901,
.23864773785017498464178231643018079921600, .7876923076923076923076923076923076923077,
.24171993688714515924331749374687206000090, .7852760736196319018404907975460122699387,
.24478272641769091566565919038112042471760, .7828746177370030581039755351681957186544,
.24783616390458124145723672882013488560910, .7804878048780487804878048780487804878049,
.25088030628580937353433455427875742316250, .7781155015197568389057750759878419452888,
.25391520998096339667426946107298135757450, .7757575757575757575757575757575757575758,
.25694093089750041913887912414793390780680, .7734138972809667673716012084592145015106,
.25995752443692604627401010475296061486000, .7710843373493975903614457831325301204819,
.26296504550088134477547896494797896593800, .7687687687687687687687687687687687687688,
.26596354849713793599974565040611196309330, .7664670658682634730538922155688622754491,
.26895308734550393836570947314612567424780, .7641791044776119402985074626865671641791,
.27193371548364175804834985683555714786050, .7619047619047619047619047619047619047619,
.27490548587279922676529508862586226314300, .7596439169139465875370919881305637982196,
.27786845100345625159121709657483734190480, .7573964497041420118343195266272189349112,
.28082266290088775395616949026589281857030, .7551622418879056047197640117994100294985,
.28376817313064456316240580235898960381750, .7529411764705882352941176470588235294118,
.28670503280395426282112225635501090437180, .7507331378299120234604105571847507331378,
.28963329258304265634293983566749375313530, .7485380116959064327485380116959064327485,
.29255300268637740579436012922087684273730, .7463556851311953352769679300291545189504,
.29546421289383584252163927885703742504130, .7441860465116279069767441860465116279070,
.29836697255179722709783618483925238251680, .7420289855072463768115942028985507246377,
.30126133057816173455023545102449133992200, .7398843930635838150289017341040462427746,
.30414733546729666446850615102448500692850, .7377521613832853025936599423631123919308,
.30702503529491181888388950937951449304830, .7356321839080459770114942528735632183908,
.30989447772286465854207904158101882785550, .7335243553008595988538681948424068767908,
.31275571000389684739317885942000430077330, .7314285714285714285714285714285714285714,
.31560877898630329552176476681779604405180, .7293447293447293447293447293447293447293,
.31845373111853458869546784626436419785030, .7272727272727272727272727272727272727273,
.32129061245373424782201254856772720813750, .7252124645892351274787535410764872521246,
.32411946865421192853773391107097268104550, .7231638418079096045197740112994350282486,
.32694034499585328257253991068864706903700, .7211267605633802816901408450704225352113,
.32975328637246797969240219572384376078850, .7191011235955056179775280898876404494382,
.33255833730007655635318997155991382896900, .7170868347338935574229691876750700280112,
.33535554192113781191153520921943709254280, .7150837988826815642458100558659217877095,
.33814494400871636381467055798566434532400, .7130919220055710306406685236768802228412,
.34092658697059319283795275623560883104800, .7111111111111111111111111111111111111111,
.34370051385331840121395430287520866841080, .7091412742382271468144044321329639889197,
.34646676734620857063262633346312213689100, .7071823204419889502762430939226519337017,
.34922538978528827602332285096053965389730, .7052341597796143250688705234159779614325,
.35197642315717814209818925519357435405250, .7032967032967032967032967032967032967033,
.35471990910292899856770532096561510115850, .7013698630136986301369863013698630136986,
.35745588892180374385176833129662554711100, .6994535519125683060109289617486338797814,
.36018440357500774995358483465679455548530, .6975476839237057220708446866485013623978,
.36290549368936841911903457003063522279280, .6956521739130434782608695652173913043478,
.36561919956096466943762379742111079394830, .6937669376693766937669376693766937669377,
.36832556115870762614150635272380895912650, .6918918918918918918918918918918918918919,
.37102461812787262962487488948681857436900, .6900269541778975741239892183288409703504,
.37371640979358405898480555151763837784530, .6881720430107526881720430107526881720430,
.37640097516425302659470730759494472295050, .6863270777479892761394101876675603217158,
.37907835293496944251145919224654790014030, .6844919786096256684491978609625668449198,
.38174858149084833769393299007788300514230, .6826666666666666666666666666666666666667,
.38441169891033200034513583887019194662580, .6808510638297872340425531914893617021277,
.38706774296844825844488013899535872042180, .6790450928381962864721485411140583554377,
.38971675114002518602873692543653305619950, .6772486772486772486772486772486772486772,
.39235876060286384303665840889152605086580, .6754617414248021108179419525065963060686,
.39499380824086893770896722344332374632350, .6736842105263157894736842105263157894737,
.39762193064713846624158577469643205404280, .6719160104986876640419947506561679790026,
.40024316412701266276741307592601515352730, .6701570680628272251308900523560209424084,
.40285754470108348090917615991202183067800, .6684073107049608355091383812010443864230,
.40546510810816432934799991016916465014230, .6666666666666666666666666666666666666667,
.40806588980822172674223224930756259709600, .6649350649350649350649350649350649350649,
.41065992498526837639616360320360399782650, .6632124352331606217616580310880829015544,
.41324724855021932601317757871584035456180, .6614987080103359173126614987080103359173,
.41582789514371093497757669865677598863850, .6597938144329896907216494845360824742268,
.41840189913888381489925905043492093682300, .6580976863753213367609254498714652956298,
.42096929464412963239894338585145305842150, .6564102564102564102564102564102564102564,
.42353011550580327293502591601281892508280, .6547314578005115089514066496163682864450,
.42608439531090003260516141381231136620050, .6530612244897959183673469387755102040816,
.42863216738969872610098832410585600882780, .6513994910941475826972010178117048346056,
.43117346481837132143866142541810404509300, .6497461928934010152284263959390862944162,
.43370832042155937902094819946796633303180, .6481012658227848101265822784810126582278,
.43623676677491801667585491486534010618930, .6464646464646464646464646464646464646465,
.43875883620762790027214350629947148263450, .6448362720403022670025188916876574307305,
.44127456080487520440058801796112675219780, .6432160804020100502512562814070351758794,
.44378397241030093089975139264424797147500, .6416040100250626566416040100250626566416,
.44628710262841947420398014401143882423650, .6400000000000000000000000000000000000000,
.44878398282700665555822183705458883196130, .6384039900249376558603491271820448877805,
.45127464413945855836729492693848442286250, .6368159203980099502487562189054726368159,
.45375911746712049854579618113348260521900, .6352357320099255583126550868486352357320,
.45623743348158757315857769754074979573500, .6336633663366336633663366336633663366337,
.45870962262697662081833982483658473938700, .6320987654320987654320987654320987654321,
.46117571512217014895185229761409573256980, .6305418719211822660098522167487684729064,
.46363574096303250549055974261136725544930, .6289926289926289926289926289926289926290,
.46608972992459918316399125615134835243230, .6274509803921568627450980392156862745098,
.46853771156323925639597405279346276074650, .6259168704156479217603911980440097799511,
.47097971521879100631480241645476780831830, .6243902439024390243902439024390243902439,
.47341577001667212165614273544633761048330, .6228710462287104622871046228710462287105,
.47584590486996386493601107758877333253630, .6213592233009708737864077669902912621359,
.47827014848147025860569669930555392056700, .6198547215496368038740920096852300242131,
.48068852934575190261057286988943815231330, .6183574879227053140096618357487922705314,
.48310107575113581113157579238759353756900, .6168674698795180722891566265060240963855,
.48550781578170076890899053978500887751580, .6153846153846153846153846153846153846154,
.48790877731923892879351001283794175833480, .6139088729016786570743405275779376498801,
.49030398804519381705802061333088204264650, .6124401913875598086124401913875598086124,
.49269347544257524607047571407747454941280, .6109785202863961813842482100238663484487,
.49507726679785146739476431321236304938800, .6095238095238095238095238095238095238095,
.49745538920281889838648226032091770321130, .6080760095011876484560570071258907363420,
.49982786955644931126130359189119189977650, .6066350710900473933649289099526066350711,
.50219473456671548383667413872899487614650, .6052009456264775413711583924349881796690,
.50455601075239520092452494282042607665050, .6037735849056603773584905660377358490566,
.50691172444485432801997148999362252652650, .6023529411764705882352941176470588235294,
.50926190178980790257412536448100581765150, .6009389671361502347417840375586854460094,
.51160656874906207391973111953120678663250, .5995316159250585480093676814988290398126,
.51394575110223428282552049495279788970950, .5981308411214953271028037383177570093458,
.51627947444845445623684554448118433356300, .5967365967365967365967365967365967365967,
.51860776420804555186805373523384332656850, .5953488372093023255813953488372093023256,
.52093064562418522900344441950437612831600, .5939675174013921113689095127610208816705,
.52324814376454775732838697877014055848100, .5925925925925925925925925925925925925926,
.52556028352292727401362526507000438869000, .5912240184757505773672055427251732101617,
.52786708962084227803046587723656557500350, .5898617511520737327188940092165898617512,
.53016858660912158374145519701414741575700, .5885057471264367816091954022988505747126,
.53246479886947173376654518506256863474850, .5871559633027522935779816513761467889908,
.53475575061602764748158733709715306758900, .5858123569794050343249427917620137299771,
.53704146589688361856929077475797384977350, .5844748858447488584474885844748858447489,
.53932196859560876944783558428753167390800, .5831435079726651480637813211845102505695,
.54159728243274429804188230264117009937750, .5818181818181818181818181818181818181818,
.54386743096728351609669971367111429572100, .5804988662131519274376417233560090702948,
.54613243759813556721383065450936555862450, .5791855203619909502262443438914027149321,
.54839232556557315767520321969641372561450, .5778781038374717832957110609480812641084,
.55064711795266219063194057525834068655950, .5765765765765765765765765765765765765766,
.55289683768667763352766542084282264113450, .5752808988764044943820224719101123595506,
.55514150754050151093110798683483153581600, .5739910313901345291479820627802690582960,
.55738115013400635344709144192165695130850, .5727069351230425055928411633109619686801,
.55961578793542265941596269840374588966350, .5714285714285714285714285714285714285714,
.56184544326269181269140062795486301183700, .5701559020044543429844097995545657015590,
.56407013828480290218436721261241473257550, .5688888888888888888888888888888888888889,
.56628989502311577464155334382667206227800, .5676274944567627494456762749445676274945,
.56850473535266865532378233183408156037350, .5663716814159292035398230088495575221239,
.57071468100347144680739575051120482385150, .5651214128035320088300220750551876379691,
.57291975356178548306473885531886480748650, .5638766519823788546255506607929515418502,
.57511997447138785144460371157038025558000, .5626373626373626373626373626373626373626,
.57731536503482350219940144597785547375700, .5614035087719298245614035087719298245614,
.57950594641464214795689713355386629700650, .5601750547045951859956236323851203501094,
.58169173963462239562716149521293118596100, .5589519650655021834061135371179039301310,
.58387276558098266665552955601015128195300, .5577342047930283224400871459694989106754,
.58604904500357812846544902640744112432000, .5565217391304347826086956521739130434783,
.58822059851708596855957011939608491957200, .5553145336225596529284164859002169197397,
.59038744660217634674381770309992134571100, .5541125541125541125541125541125541125541,
.59254960960667157898740242671919986605650, .5529157667386609071274298056155507559395,
.59470710774669277576265358220553025603300, .5517241379310344827586206896551724137931,
.59685996110779382384237123915227130055450, .5505376344086021505376344086021505376344,
.59900818964608337768851242799428291618800, .5493562231759656652360515021459227467811,
.60115181318933474940990890900138765573500, .5481798715203426124197002141327623126338,
.60329085143808425240052883964381180703650, .5470085470085470085470085470085470085470,
.60542532396671688843525771517306566238400, .5458422174840085287846481876332622601279,
.60755525022454170969155029524699784815300, .5446808510638297872340425531914893617021,
.60968064953685519036241657886421307921400, .5435244161358811040339702760084925690021,
.61180154110599282990534675263916142284850, .5423728813559322033898305084745762711864,
.61391794401237043121710712512140162289150, .5412262156448202959830866807610993657505,
.61602987721551394351138242200249806046500, .5400843881856540084388185654008438818565,
.61813735955507864705538167982012964785100, .5389473684210526315789473684210526315789,
.62024040975185745772080281312810257077200, .5378151260504201680672268907563025210084,
.62233904640877868441606324267922900617100, .5366876310272536687631027253668763102725,
.62443328801189346144440150965237990021700, .5355648535564853556485355648535564853556,
.62652315293135274476554741340805776417250, .5344467640918580375782881002087682672234,
.62860865942237409420556559780379757285100, .5333333333333333333333333333333333333333,
.63068982562619868570408243613201193511500, .5322245322245322245322245322245322245322,
.63276666957103777644277897707070223987100, .5311203319502074688796680497925311203320,
.63483920917301017716738442686619237065300, .5300207039337474120082815734989648033126,
.63690746223706917739093569252872839570050, .5289256198347107438016528925619834710744,
.63897144645792069983514238629140891134750, .5278350515463917525773195876288659793814,
.64103117942093124081992527862894348800200, .5267489711934156378600823045267489711934,
.64308667860302726193566513757104985415950, .5256673511293634496919917864476386036961,
.64513796137358470073053240412264131009600, .5245901639344262295081967213114754098361,
.64718504499530948859131740391603671014300, .5235173824130879345603271983640081799591,
.64922794662510974195157587018911726772800, .5224489795918367346938775510204081632653,
.65126668331495807251485530287027359008800, .5213849287169042769857433808553971486762,
.65330127201274557080523663898929953575150, .5203252032520325203252032520325203252033,
.65533172956312757406749369692988693714150, .5192697768762677484787018255578093306288,
.65735807270835999727154330685152672231200, .5182186234817813765182186234817813765182,
.65938031808912778153342060249997302889800, .5171717171717171717171717171717171717172,
.66139848224536490484126716182800009846700, .5161290322580645161290322580645161290323,
.66341258161706617713093692145776003599150, .5150905432595573440643863179074446680080,
.66542263254509037562201001492212526500250, .5140562248995983935742971887550200803213,
.66742865127195616370414654738851822912700, .5130260521042084168336673346693386773547,
.66943065394262923906154583164607174694550, .5120000000000000000000000000000000000000,
.67142865660530226534774556057527661323550, .5109780439121756487025948103792415169661,
.67342267521216669923234121597488410770900, .5099601593625498007968127490039840637450,
.67541272562017662384192817626171745359900, .5089463220675944333996023856858846918489,
.67739882359180603188519853574689477682100, .5079365079365079365079365079365079365079,
.67938098479579733801614338517538271844400, .5069306930693069306930693069306930693069,
.68135922480790300781450241629499942064300, .5059288537549407114624505928853754940711,
.68333355911162063645036823800182901322850, .5049309664694280078895463510848126232742,
.68530400309891936760919861626462079584600, .5039370078740157480314960629921259842520,
.68727057207096020619019327568821609020250, .5029469548133595284872298624754420432220,
.68923328123880889251040571252815425395950, .5019607843137254901960784313725490196078,
.69314718055994530941723212145818, 5.0e-01,
};
const double* getLogTab64f()
{
return logTab;
}
const float* getLogTab32f()
{
static float CV_DECL_ALIGNED(64) logTab_f[(LOGTAB_MASK+1)*2];
static std::atomic<bool> logTab_f_initialized(false);
if (!logTab_f_initialized.load())
{
for (int j = 0; j < (LOGTAB_MASK+1)*2; j++)
logTab_f[j] = (float)logTab[j];
logTab_f_initialized = true;
}
return logTab_f;
}
}} // namespace
/* End of file. */
modules/core/src/mathfuncs.hpp
+2 -6
View File
@@ -5,11 +5,7 @@
#ifndef OPENCV_CORE_SRC_MATHFUNCS_HPP
#define OPENCV_CORE_SRC_MATHFUNCS_HPP
namespace cv { namespace details {
const double* getExpTab64f();
const float* getExpTab32f();
const double* getLogTab64f();
const float* getLogTab32f();
}} // namespace
// (the exp/log table kernels and their tables are gone - cv::hal::exp32f & co now fall back
// to the element-wise engine's vector kernels; see mathfuncs_core.dispatch.cpp)
#endif // OPENCV_CORE_SRC_MATHFUNCS_HPP
modules/core/src/mathfuncs_core.dispatch.cpp
+9 -8
View File
@@ -3,6 +3,7 @@
// of this distribution and at http://opencv.org/license.html.
#include "precomp.hpp"
#include "arithm_expr.hpp" // ew::mathSpanEngine - the engine fallback for exp/log
#include "mathfuncs_core.simd.hpp"
#include "mathfuncs_core.simd_declarations.hpp" // defines CV_CPU_DISPATCH_MODES_ALL=AVX2,...,BASELINE based on CMakeLists.txt content
@@ -151,8 +152,8 @@ void exp32f(const float *src, float *dst, int n)
CALL_HAL(exp32f, cv_hal_exp32f, src, dst, n);
CV_CPU_DISPATCH(exp32f, (src, dst, n),
CV_CPU_DISPATCH_MODES_ALL);
ew::mathSpanEngine(ew::OP_EXP, CV_32F, src, dst, n); // the engine's vector kernel (the old
// table implementation is removed)
}
void exp64f(const double *src, double *dst, int n)
@@ -161,8 +162,8 @@ void exp64f(const double *src, double *dst, int n)
CALL_HAL(exp64f, cv_hal_exp64f, src, dst, n);
CV_CPU_DISPATCH(exp64f, (src, dst, n),
CV_CPU_DISPATCH_MODES_ALL);
ew::mathSpanEngine(ew::OP_EXP, CV_64F, src, dst, n); // the engine's vector kernel (the old
// table implementation is removed)
}
void log32f(const float *src, float *dst, int n)
@@ -171,8 +172,8 @@ void log32f(const float *src, float *dst, int n)
CALL_HAL(log32f, cv_hal_log32f, src, dst, n);
CV_CPU_DISPATCH(log32f, (src, dst, n),
CV_CPU_DISPATCH_MODES_ALL);
ew::mathSpanEngine(ew::OP_LOG, CV_32F, src, dst, n); // the engine's vector kernel (the old
// table implementation is removed)
}
void log64f(const double *src, double *dst, int n)
@@ -181,8 +182,8 @@ void log64f(const double *src, double *dst, int n)
CALL_HAL(log64f, cv_hal_log64f, src, dst, n);
CV_CPU_DISPATCH(log64f, (src, dst, n),
CV_CPU_DISPATCH_MODES_ALL);
ew::mathSpanEngine(ew::OP_LOG, CV_64F, src, dst, n); // the engine's vector kernel (the old
// table implementation is removed)
}
//=============================================================================
modules/core/src/mathfuncs_core.simd.hpp
-420
View File
@@ -22,10 +22,6 @@ void invSqrt32f(const float* src, float* dst, int len);
void invSqrt64f(const double* src, double* dst, int len);
void sqrt32f(const float* src, float* dst, int len);
void sqrt64f(const double* src, double* dst, int len);
void exp32f(const float *src, float *dst, int n);
void exp64f(const double *src, double *dst, int n);
void log32f(const float *src, float *dst, int n);
void log64f(const double *src, double *dst, int n);
float fastAtan2(float y, float x);
#ifndef CV_CPU_OPTIMIZATION_DECLARATIONS_ONLY
@@ -638,422 +634,6 @@ void log64f(const double *src, double *dst, int n)
////////////////////////////////////// EXP /////////////////////////////////////
#define EXPTAB_SCALE 6
#define EXPTAB_MASK ((1 << EXPTAB_SCALE) - 1)
#define EXPPOLY_32F_A0 .9670371139572337719125840413672004409288e-2
// the code below uses _mm_cast* intrinsics, which are not available on VS2005
#if (defined _MSC_VER && _MSC_VER < 1500) || \
(!defined __APPLE__ && defined __GNUC__ && __GNUC__*100 + __GNUC_MINOR__ < 402)
#undef CV_SSE2
#define CV_SSE2 0
#endif
static const double exp_prescale = 1.4426950408889634073599246810019 * (1 << EXPTAB_SCALE);
static const double exp_postscale = 1./(1 << EXPTAB_SCALE);
static const double exp_max_val = 3000.*(1 << EXPTAB_SCALE); // log10(DBL_MAX) < 3000
void exp32f( const float *_x, float *y, int n )
{
CV_INSTRUMENT_REGION();
const float* const expTab_f = cv::details::getExpTab32f();
const float
A4 = (float)(1.000000000000002438532970795181890933776 / EXPPOLY_32F_A0),
A3 = (float)(.6931471805521448196800669615864773144641 / EXPPOLY_32F_A0),
A2 = (float)(.2402265109513301490103372422686535526573 / EXPPOLY_32F_A0),
A1 = (float)(.5550339366753125211915322047004666939128e-1 / EXPPOLY_32F_A0);
int i = 0;
const Cv32suf* x = (const Cv32suf*)_x;
float minval = (float)(-exp_max_val/exp_prescale);
float maxval = (float)(exp_max_val/exp_prescale);
float postscale = (float)exp_postscale;
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int VECSZ = VTraits<v_float32>::vlanes();
const v_float32 vprescale = vx_setall_f32((float)exp_prescale);
const v_float32 vpostscale = vx_setall_f32((float)exp_postscale);
const v_float32 vminval = vx_setall_f32(minval);
const v_float32 vmaxval = vx_setall_f32(maxval);
const v_float32 vA1 = vx_setall_f32((float)A1);
const v_float32 vA2 = vx_setall_f32((float)A2);
const v_float32 vA3 = vx_setall_f32((float)A3);
const v_float32 vA4 = vx_setall_f32((float)A4);
const v_int32 vidxmask = vx_setall_s32(EXPTAB_MASK);
bool y_aligned = (size_t)(void*)y % 32 == 0;
for( ; i < n; i += VECSZ*2 )
{
if( i + VECSZ*2 > n )
{
if( i == 0 || _x == y )
break;
i = n - VECSZ*2;
y_aligned = false;
}
v_float32 xf0 = vx_load(&x[i].f), xf1 = vx_load(&x[i + VECSZ].f);
xf0 = v_min(v_max(xf0, vminval), vmaxval);
xf1 = v_min(v_max(xf1, vminval), vmaxval);
xf0 = v_mul(xf0, vprescale);
xf1 = v_mul(xf1, vprescale);
v_int32 xi0 = v_round(xf0);
v_int32 xi1 = v_round(xf1);
xf0 = v_mul(v_sub(xf0, v_cvt_f32(xi0)), vpostscale);
xf1 = v_mul(v_sub(xf1, v_cvt_f32(xi1)), vpostscale);
v_float32 yf0 = v_lut(expTab_f, v_and(xi0, vidxmask));
v_float32 yf1 = v_lut(expTab_f, v_and(xi1, vidxmask));
v_int32 v0 = vx_setzero_s32(), v127 = vx_setall_s32(127), v255 = vx_setall_s32(255);
xi0 = v_min(v_max(v_add(v_shr<6>(xi0), v127), v0), v255);
xi1 = v_min(v_max(v_add(v_shr<6>(xi1), v127), v0), v255);
yf0 = v_mul(yf0, v_reinterpret_as_f32(v_shl<23>(xi0)));
yf1 = v_mul(yf1, v_reinterpret_as_f32(v_shl<23>(xi1)));
v_float32 zf0 = v_add(xf0, vA1);
v_float32 zf1 = v_add(xf1, vA1);
zf0 = v_fma(zf0, xf0, vA2);
zf1 = v_fma(zf1, xf1, vA2);
zf0 = v_fma(zf0, xf0, vA3);
zf1 = v_fma(zf1, xf1, vA3);
zf0 = v_fma(zf0, xf0, vA4);
zf1 = v_fma(zf1, xf1, vA4);
zf0 = v_mul(zf0, yf0);
zf1 = v_mul(zf1, yf1);
if( y_aligned )
{
v_store_aligned(y + i, zf0);
v_store_aligned(y + i + VECSZ, zf1);
}
else
{
v_store(y + i, zf0);
v_store(y + i + VECSZ, zf1);
}
}
vx_cleanup();
#endif
for( ; i < n; i++ )
{
float x0 = x[i].f;
x0 = std::min(std::max(x0, minval), maxval);
x0 *= (float)exp_prescale;
Cv32suf buf;
int xi = saturate_cast<int>(x0);
x0 = (x0 - xi)*postscale;
int t = (xi >> EXPTAB_SCALE) + 127;
t = !(t & ~255) ? t : t < 0 ? 0 : 255;
buf.i = t << 23;
y[i] = buf.f * expTab_f[xi & EXPTAB_MASK] * ((((x0 + A1)*x0 + A2)*x0 + A3)*x0 + A4);
}
}
void exp64f( const double *_x, double *y, int n )
{
CV_INSTRUMENT_REGION();
const double* const expTab = cv::details::getExpTab64f();
const double
A5 = .99999999999999999998285227504999 / EXPPOLY_32F_A0,
A4 = .69314718055994546743029643825322 / EXPPOLY_32F_A0,
A3 = .24022650695886477918181338054308 / EXPPOLY_32F_A0,
A2 = .55504108793649567998466049042729e-1 / EXPPOLY_32F_A0,
A1 = .96180973140732918010002372686186e-2 / EXPPOLY_32F_A0,
A0 = .13369713757180123244806654839424e-2 / EXPPOLY_32F_A0;
int i = 0;
const Cv64suf* x = (const Cv64suf*)_x;
double minval = (-exp_max_val/exp_prescale);
double maxval = (exp_max_val/exp_prescale);
#if (CV_SIMD_64F || CV_SIMD_SCALABLE_64F)
const int VECSZ = VTraits<v_float64>::vlanes();
const v_float64 vprescale = vx_setall_f64(exp_prescale);
const v_float64 vpostscale = vx_setall_f64(exp_postscale);
const v_float64 vminval = vx_setall_f64(minval);
const v_float64 vmaxval = vx_setall_f64(maxval);
const v_float64 vA1 = vx_setall_f64(A1);
const v_float64 vA2 = vx_setall_f64(A2);
const v_float64 vA3 = vx_setall_f64(A3);
const v_float64 vA4 = vx_setall_f64(A4);
const v_float64 vA5 = vx_setall_f64(A5);
const v_int32 vidxmask = vx_setall_s32(EXPTAB_MASK);
bool y_aligned = (size_t)(void*)y % 32 == 0;
for( ; i < n; i += VECSZ*2 )
{
if( i + VECSZ*2 > n )
{
if( i == 0 || _x == y )
break;
i = n - VECSZ*2;
y_aligned = false;
}
v_float64 xf0 = vx_load(&x[i].f), xf1 = vx_load(&x[i + VECSZ].f);
xf0 = v_min(v_max(xf0, vminval), vmaxval);
xf1 = v_min(v_max(xf1, vminval), vmaxval);
xf0 = v_mul(xf0, vprescale);
xf1 = v_mul(xf1, vprescale);
v_int32 xi0 = v_round(xf0);
v_int32 xi1 = v_round(xf1);
xf0 = v_mul(v_sub(xf0, v_cvt_f64(xi0)), vpostscale);
xf1 = v_mul(v_sub(xf1, v_cvt_f64(xi1)), vpostscale);
v_float64 yf0 = v_lut(expTab, v_and(xi0, vidxmask));
v_float64 yf1 = v_lut(expTab, v_and(xi1, vidxmask));
v_int32 v0 = vx_setzero_s32(), v1023 = vx_setall_s32(1023), v2047 = vx_setall_s32(2047);
xi0 = v_min(v_max(v_add(v_shr<6>(xi0), v1023), v0), v2047);
xi1 = v_min(v_max(v_add(v_shr<6>(xi1), v1023), v0), v2047);
v_int64 xq0, xq1, dummy;
v_expand(xi0, xq0, dummy);
v_expand(xi1, xq1, dummy);
yf0 = v_mul(yf0, v_reinterpret_as_f64(v_shl<52>(xq0)));
yf1 = v_mul(yf1, v_reinterpret_as_f64(v_shl<52>(xq1)));
v_float64 zf0 = v_add(xf0, vA1);
v_float64 zf1 = v_add(xf1, vA1);
zf0 = v_fma(zf0, xf0, vA2);
zf1 = v_fma(zf1, xf1, vA2);
zf0 = v_fma(zf0, xf0, vA3);
zf1 = v_fma(zf1, xf1, vA3);
zf0 = v_fma(zf0, xf0, vA4);
zf1 = v_fma(zf1, xf1, vA4);
zf0 = v_fma(zf0, xf0, vA5);
zf1 = v_fma(zf1, xf1, vA5);
zf0 = v_mul(zf0, yf0);
zf1 = v_mul(zf1, yf1);
if( y_aligned )
{
v_store_aligned(y + i, zf0);
v_store_aligned(y + i + VECSZ, zf1);
}
else
{
v_store(y + i, zf0);
v_store(y + i + VECSZ, zf1);
}
}
vx_cleanup();
#endif
for( ; i < n; i++ )
{
double x0 = x[i].f;
x0 = std::min(std::max(x0, minval), maxval);
x0 *= exp_prescale;
Cv64suf buf;
int xi = saturate_cast<int>(x0);
x0 = (x0 - xi)*exp_postscale;
int t = (xi >> EXPTAB_SCALE) + 1023;
t = !(t & ~2047) ? t : t < 0 ? 0 : 2047;
buf.i = (int64)t << 52;
y[i] = buf.f * expTab[xi & EXPTAB_MASK] * (((((A0*x0 + A1)*x0 + A2)*x0 + A3)*x0 + A4)*x0 + A5);
}
}
#undef EXPTAB_SCALE
#undef EXPTAB_MASK
#undef EXPPOLY_32F_A0
/////////////////////////////////////////// LOG ///////////////////////////////////////
#define LOGTAB_SCALE 8
#define LOGTAB_MASK ((1 << LOGTAB_SCALE) - 1)
#define LOGTAB_TRANSLATE(tab, x, h) (((x) - 1.f)*tab[(h)+1])
static const double ln_2 = 0.69314718055994530941723212145818;
void log32f( const float *_x, float *y, int n )
{
CV_INSTRUMENT_REGION();
const float* const logTab_f = cv::details::getLogTab32f();
const int LOGTAB_MASK2_32F = (1 << (23 - LOGTAB_SCALE)) - 1;
const float
A0 = 0.3333333333333333333333333f,
A1 = -0.5f,
A2 = 1.f;
int i = 0;
const int* x = (const int*)_x;
#if (CV_SIMD || CV_SIMD_SCALABLE)
const int VECSZ = VTraits<v_float32>::vlanes();
const v_float32 vln2 = vx_setall_f32((float)ln_2);
const v_float32 v1 = vx_setall_f32(1.f);
const v_float32 vshift = vx_setall_f32(-1.f/512);
const v_float32 vA0 = vx_setall_f32(A0);
const v_float32 vA1 = vx_setall_f32(A1);
const v_float32 vA2 = vx_setall_f32(A2);
for( ; i < n; i += VECSZ )
{
if( i + VECSZ > n )
{
if( i == 0 || _x == y )
break;
i = n - VECSZ;
}
v_int32 h0 = vx_load(x + i);
v_int32 yi0 = v_sub(v_and(v_shr<23>(h0), vx_setall_s32(255)), vx_setall_s32(127));
v_int32 xi0 = v_or(v_and(h0, vx_setall_s32(LOGTAB_MASK2_32F)), vx_setall_s32(127 << 23));
h0 = v_and(v_shr<23 - 8 - 1>(h0), vx_setall_s32(((1 << 8) - 1) * 2));
v_float32 yf0, xf0;
v_lut_deinterleave(logTab_f, h0, yf0, xf0);
yf0 = v_fma(v_cvt_f32(yi0), vln2, yf0);
v_float32 delta = v_select(v_reinterpret_as_f32(v_eq(h0, vx_setall_s32(510))), vshift, vx_setall<float>(0));
xf0 = v_fma((v_sub(v_reinterpret_as_f32(xi0), v1)), xf0, delta);
v_float32 zf0 = v_fma(xf0, vA0, vA1);
zf0 = v_fma(zf0, xf0, vA2);
zf0 = v_fma(zf0, xf0, yf0);
v_store(y + i, zf0);
}
vx_cleanup();
#endif
for( ; i < n; i++ )
{
Cv32suf buf;
int i0 = x[i];
buf.i = (i0 & LOGTAB_MASK2_32F) | (127 << 23);
int idx = (i0 >> (23 - LOGTAB_SCALE - 1)) & (LOGTAB_MASK*2);
float y0 = (((i0 >> 23) & 0xff) - 127) * (float)ln_2 + logTab_f[idx];
float x0 = (buf.f - 1.f)*logTab_f[idx + 1] + (idx == 510 ? -1.f/512 : 0.f);
y[i] = ((A0*x0 + A1)*x0 + A2)*x0 + y0;
}
}
void log64f( const double *x, double *y, int n )
{
CV_INSTRUMENT_REGION();
const double* const logTab = cv::details::getLogTab64f();
const int64 LOGTAB_MASK2_64F = ((int64)1 << (52 - LOGTAB_SCALE)) - 1;
const double
A7 = 1.0,
A6 = -0.5,
A5 = 0.333333333333333314829616256247390992939472198486328125,
A4 = -0.25,
A3 = 0.2,
A2 = -0.1666666666666666574148081281236954964697360992431640625,
A1 = 0.1428571428571428769682682968777953647077083587646484375,
A0 = -0.125;
int i = 0;
#if (CV_SIMD_64F || CV_SIMD_SCALABLE_64F)
const int VECSZ = VTraits<v_float64>::vlanes();
const v_float64 vln2 = vx_setall_f64(ln_2);
const v_float64
vA0 = vx_setall_f64(A0), vA1 = vx_setall_f64(A1),
vA2 = vx_setall_f64(A2), vA3 = vx_setall_f64(A3),
vA4 = vx_setall_f64(A4), vA5 = vx_setall_f64(A5),
vA6 = vx_setall_f64(A6), vA7 = vx_setall_f64(A7);
for( ; i < n; i += VECSZ )
{
if( i + VECSZ > n )
{
if( i == 0 || x == y )
break;
i = n - VECSZ;
}
v_int64 h0 = vx_load((const int64*)x + i);
v_int32 yi0 = v_pack(v_shr<52>(h0), vx_setzero_s64());
yi0 = v_sub(v_and(yi0, vx_setall_s32(2047)), vx_setall_s32(1023));
v_int64 xi0 = v_or(v_and(h0, vx_setall_s64(LOGTAB_MASK2_64F)), vx_setall_s64((int64)1023 << 52));
h0 = v_shr<52 - LOGTAB_SCALE - 1>(h0);
v_int32 idx = v_and(v_pack(h0, h0), vx_setall_s32(((1 << 8) - 1) * 2));
v_float64 xf0, yf0;
v_lut_deinterleave(logTab, idx, yf0, xf0);
yf0 = v_fma(v_cvt_f64(yi0), vln2, yf0);
v_float64 delta = v_mul(v_cvt_f64(v_eq(idx, vx_setall_s32(510))), vx_setall_f64(1. / 512));
xf0 = v_fma(v_sub(v_reinterpret_as_f64(xi0), vx_setall_f64(1.)), xf0, delta);
v_float64 xq = v_mul(xf0, xf0);
v_float64 zf0 = v_fma(xq, vA0, vA2);
v_float64 zf1 = v_fma(xq, vA1, vA3);
zf0 = v_fma(zf0, xq, vA4);
zf1 = v_fma(zf1, xq, vA5);
zf0 = v_fma(zf0, xq, vA6);
zf1 = v_fma(zf1, xq, vA7);
zf1 = v_fma(zf1, xf0, yf0);
zf0 = v_fma(zf0, xq, zf1);
v_store(y + i, zf0);
}
#endif
for( ; i < n; i++ )
{
Cv64suf buf;
int64 i0 = ((const int64*)x)[i];
buf.i = (i0 & LOGTAB_MASK2_64F) | ((int64)1023 << 52);
int idx = (int)(i0 >> (52 - LOGTAB_SCALE - 1)) & (LOGTAB_MASK*2);
double y0 = (((int)(i0 >> 52) & 0x7ff) - 1023) * ln_2 + logTab[idx];
double x0 = (buf.f - 1.)*logTab[idx + 1] + (idx == 510 ? -1./512 : 0.);
double xq = x0*x0;
y[i] = (((A0*xq + A2)*xq + A4)*xq + A6)*xq + (((A1*xq + A3)*xq + A5)*xq + A7)*x0 + y0;
}
}
#endif // issue 7795
modules/core/src/ocl.cpp
+15 -5
View File
@@ -7253,12 +7253,21 @@ int predictOptimalVectorWidth(InputArray src1, InputArray src2, InputArray src3,
return checkOptimalVectorWidth(vectorWidths, src1, src2, src3, src4, src5, src6, src7, src8, src9, strat);
}
int checkOptimalVectorWidth(const int *vectorWidths,
InputArray src1, InputArray src2, InputArray src3,
InputArray src4, InputArray src5, InputArray src6,
InputArray src7, InputArray src8, InputArray src9,
OclVectorStrategy strat)
int checkOptimalVectorWidth([[maybe_unused]] const int *vectorWidths,
[[maybe_unused]] InputArray src1,
[[maybe_unused]] InputArray src2,
[[maybe_unused]] InputArray src3,
[[maybe_unused]] InputArray src4,
[[maybe_unused]] InputArray src5,
[[maybe_unused]] InputArray src6,
[[maybe_unused]] InputArray src7,
[[maybe_unused]] InputArray src8,
[[maybe_unused]] InputArray src9,
[[maybe_unused]] OclVectorStrategy strat)
{
#ifdef __APPLE__
return 1;
#else
CV_Assert(vectorWidths);
int ref_type = src1.type();
@@ -7285,6 +7294,7 @@ int checkOptimalVectorWidth(const int *vectorWidths,
int kercn = *std::min_element(kercns.begin(), kercns.end());
return kercn;
#endif
}
int predictOptimalVectorWidthMax(InputArray src1, InputArray src2, InputArray src3,
modules/core/src/precomp.hpp
+149 -2
View File
@@ -261,8 +261,12 @@ typedef void (*BinaryFuncC)(const uchar* src1, size_t step1,
uchar* dst, size_t step, int width, int height,
void*);
BinaryFunc getConvertFunc(int sdepth, int ddepth);
BinaryFunc getConvertScaleFunc(int sdepth, int ddepth);
// Exported so the new element-wise expression engine can reuse the already-optimized,
// CPU-dispatched convert / convert-scale kernels through a thin ElemwiseFunc adapter,
// instead of re-implementing the whole cast matrix. (Prototype: declarations are mirrored
// engine-side; relocate into a public core header at integration time.)
CV_EXPORTS BinaryFunc getConvertFunc(int sdepth, int ddepth);
CV_EXPORTS BinaryFunc getConvertScaleFunc(int sdepth, int ddepth);
BinaryFunc getCopyMaskFunc(size_t esz);
/* default memory block for sparse array elements */
@@ -328,6 +332,149 @@ inline bool checkScalar(InputArray sc, int atype, _InputArray::KindFlag sckind,
(sz == Size(1, 4) && sc.type() == CV_64F && cn <= 4);
}
// New element-wise engine scalar handling. A genuine number / Scalar / Vec / Matx operand to an
// arithmetic op arrives via _InputArray::MATX (its data is inline in the caller's object;
// getObj() points straight at it). In addition, the EXACT Scalar materialization that the
// python/java bindings and operator-(Mat, Matx) produce - a 2-D 4x1 CV_64F single-channel Mat or
// UMat - is a scalar UNCONDITIONALLY: by coincidence it can be broadcast-COMPATIBLE with the array
// (a 4-row array, a 1-D array make (4,1) legal numpy-wise), and the 4.x per-channel-scalar
// semantics must win there for binding users. Any other real Mat/UMat rides normal broadcasting
// (but see isScalarLikeMat below for the shape-incompatible compat fallback).
inline bool isScalarArg(const _InputArray& sc, int cn)
{
const _InputArray::KindFlag kind = sc.kind();
if (kind == _InputArray::MATX)
{
Size sz = sc.getSz();
int scn0 = sz.width * sz.height;
// A genuine scalar is a 1D MATX (a Vec/Scalar/number: one of width/height is 1). A 2D MATX
// (e.g. a Matx33) is a real matrix operand and rides broadcasting - never a scalar.
if (scn0 != sz.width + sz.height - 1)
return false;
// Per-channel match (incl. Vec<_,N> for an N-channel array), a 4-elem Scalar on a <4-channel
// array, or a single broadcast value. No 4-channel cap: a multichannel scalar rides as a 0-dim
// per-channel CONST over the caller's data (not squeezed into a 4-slot Scalar).
return scn0 == cn || (cn < 4 && scn0 == 4) || scn0 == 1;
}
// the bindings-style Scalar column. dims must be exactly 2: a 1-D [4] CV_64F array is an honest
// broadcast operand. Direct field reads (no _InputArray getter dispatch) - this runs on EVERY
// engine call with Mat operands, and `rows == 4` alone rejects almost every real array.
if (kind == _InputArray::MAT)
{
const Mat& m = *(const Mat*)sc.getObj();
return m.rows == 4 && m.cols == 1 && cn <= 4 && m.dims == 2 &&
m.type() == CV_64F && m.isContinuous();
}
if (kind == _InputArray::UMAT)
{
const UMat& m = *(const UMat*)sc.getObj();
return m.rows == 4 && m.cols == 1 && cn <= 4 && m.dims == 2 &&
m.type() == CV_64F && m.isContinuous();
}
return false;
}
// The remaining old arithm_op checkScalar geometry: a real Mat/UMat that LOOKS like a scalar - 1x1
// or a 1xcn/cnx1 vector (the 4x1 CV_64F column is handled unconditionally by isScalarArg above).
// arithm_op treats such an operand as a per-channel scalar ONLY as a fallback, when the shapes are
// not broadcast-compatible - a call that is valid under numpy rules keeps its numpy meaning, one
// that would throw gets the 4.x scalar semantics instead.
inline bool isScalarLikeMat(const _InputArray& sc, int cn)
{
// direct field reads, like isScalarArg: this probe also runs per engine call (see arithm_op's
// compat fallback). `rows != 1 && cols != 1` exits in two inline compares for ordinary arrays.
int rows, cols, nval;
bool continuous;
const _InputArray::KindFlag kind = sc.kind();
if (kind == _InputArray::MAT)
{
const Mat& m = *(const Mat*)sc.getObj();
if ((m.rows != 1 && m.cols != 1) || m.dims > 2) return false;
rows = m.rows; cols = m.cols; nval = rows * cols * m.channels(); continuous = m.isContinuous();
}
else if (kind == _InputArray::UMAT)
{
const UMat& m = *(const UMat*)sc.getObj();
if ((m.rows != 1 && m.cols != 1) || m.dims > 2) return false;
rows = m.rows; cols = m.cols; nval = rows * cols * m.channels(); continuous = m.isContinuous();
}
else
return false;
if (!continuous || nval > 4)
return false;
return (rows == 1 && cols == 1) || (rows == cn && cols == 1) || (rows == 1 && cols == cn);
}
// The size of the caller-provided stack buffer for scalarArgElems (4 slots of the widest depth).
// A scalar is <= 4 values by contract, so it NEVER touches the heap.
enum { EW_SCALAR_BUF_SIZE = 4 * sizeof(double) };
// A scalar operand's raw payload: the MATX inline storage or the Mat data, both returned in place.
// A UMAT scalar's values are copied device->host into `scbuf` (>= EW_SCALAR_BUF_SIZE bytes, on the
// CALLER's stack - no heap, no UMat::getMat mapping machinery for 32 bytes of data).
// p/d receive the data pointer and depth; returns the value count (elems x channels).
inline int scalarArgElems(const _InputArray& sc, const uchar*& p, int& d, uchar* scbuf)
{
const _InputArray::KindFlag kind = sc.kind();
if (kind == _InputArray::MAT)
{
const Mat& m = *(const Mat*)sc.getObj();
p = m.data;
d = m.depth();
return (int)m.total() * m.channels();
}
if (kind == _InputArray::UMAT)
{
const UMat& u = *(const UMat*)sc.getObj();
d = u.depth();
int n = (int)u.total() * u.channels();
CV_Assert(n * (int)CV_ELEM_SIZE1(d) <= (int)EW_SCALAR_BUF_SIZE);
Mat header(u.dims, u.size.p, u.type(), scbuf); // header over the caller's stack buffer
u.copyTo(header); // create() is a no-op (exact match) -> the
p = scbuf; // copy lands straight in scbuf
return n;
}
p = (const uchar*)sc.getObj();
d = sc.depth();
Size sz = sc.getSz();
return sz.width * sz.height;
}
// Read one element of depth `d` at p as a double (no Mat, no convertTo, no dispatcher).
inline double elemToDouble(int d, const uchar* p)
{
switch (d)
{
case CV_8U: return *(const uchar*)p;
case CV_8S: return *(const schar*)p;
case CV_16U: return *(const ushort*)p;
case CV_16S: return *(const short*)p;
case CV_32U: return *(const unsigned*)p;
case CV_32S: return *(const int*)p;
case CV_64U: return (double)*(const uint64_t*)p;
case CV_64S: return (double)*(const int64_t*)p;
case CV_16F: return (float)*(const hfloat*)p;
case CV_16BF: return (float)*(const bfloat*)p;
case CV_32F: return *(const float*)p;
case CV_64F: return *(const double*)p;
default: CV_Error(Error::StsUnsupportedFormat, "unsupported scalar depth");
}
}
// Extract a scalar operand's values (see isScalarArg) as up to 4 doubles, straight from its
// storage. Returns the element count.
inline int readScalarArg(const _InputArray& sc, Scalar& out)
{
out = Scalar(); // unused channels stay 0 (independent of the caller's Scalar)
const uchar* p; int d; uchar scbuf[EW_SCALAR_BUF_SIZE];
int n = scalarArgElems(sc, p, d, scbuf);
CV_Assert(n <= 4); // Scalar holds 4 slots; isScalarArg admits more only for MATX
size_t esz = CV_ELEM_SIZE1(d);
for (int i = 0; i < n; i++)
out[i] = elemToDouble(d, p + (size_t)i * esz);
return n;
}
void convertAndUnrollScalar( const Mat& sc, int buftype, uchar* scbuf, size_t blocksize );
#ifdef CV_COLLECT_IMPL_DATA
modules/core/test/test_arithm.cpp
+5 -1
View File
@@ -2514,10 +2514,14 @@ TEST(Compare, empty)
TEST(Compare, regression_8999)
{
// Issue #8999 predates broadcasting element-wise ops: comparing a 4x1 array against a 1x1 operand
// used to throw (both look like a Scalar). It now broadcasts the 1x1 operand across the 4x1 array.
Mat_<double> A(4,1); A << 1, 3, 2, 4;
Mat_<double> B(1,1); B << 2;
Mat C;
EXPECT_THROW(cv::compare(A, B, C, CMP_LT), cv::Exception);
cv::compare(A, B, C, CMP_LT);
Mat expected = (Mat_<uchar>(4,1) << 255, 0, 0, 0); // A < 2
EXPECT_EQ(0, cvtest::norm(C, expected, NORM_INF));
}
TEST(Compare, regression_16F_do_not_crash)
modules/core/test/test_arithm_expr.cpp
+512
View File
@@ -0,0 +1,512 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// Tests for the public cv::texpr() string frontend. Exercises placeholders, operator precedence,
// function calls, type casts, assignments and tuple (multi-) outputs. Limited to ops with kernels
// today (arithmetic / cast / pow / min / max / absdiff).
#include "test_precomp.hpp"
#include "../src/arithm_expr.hpp" // ew::absdiffResultDepth - the engine's absdiff auto-type rule
namespace opencv_test { namespace {
static Mat expr1(const String& e, const std::vector<Mat>& in)
{
std::vector<Mat> out;
cv::texpr(e, in, out);
return out[0];
}
TEST(Core_TExpr, add)
{
Mat a(12, 15, CV_32F), b(12, 15, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
Mat got = expr1("{0} + {1}", { a, b });
Mat exp; cv::add(a, b, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3);
}
// Built-in binary functions min/max/absdiff parsed and dispatched through emitBinary.
TEST(Core_TExpr, minmax_absdiff)
{
Mat a(18, 21, CV_8U), b(18, 21, CV_8U);
theRNG().fill(a, RNG::UNIFORM, 0, 255);
theRNG().fill(b, RNG::UNIFORM, 0, 255);
Mat gmin = expr1("min({0}, {1})", { a, b });
Mat gmax = expr1("max({0}, {1})", { a, b });
Mat gabs = expr1("absdiff({0}, {1})", { a, b });
Mat emin, emax, eabs;
cv::min(a, b, emin); cv::max(a, b, emax); cv::absdiff(a, b, eabs);
EXPECT_EQ(0, cvtest::norm(gmin, emin, NORM_INF));
EXPECT_EQ(0, cvtest::norm(gmax, emax, NORM_INF));
EXPECT_EQ(0, cvtest::norm(gabs, eabs, NORM_INF));
}
// Operator precedence: '*' binds tighter than '+', unary minus on a literal.
TEST(Core_TExpr, addweighted_precedence)
{
Mat a(20, 16, CV_32F), b(20, 16, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
Mat got = expr1("{0} * 2.5 + {1} * -1.5 + 7", { a, b });
Mat exp; cv::addWeighted(a, 2.5, b, -1.5, 7.0, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3);
}
// Named temporary via ';' assignment.
TEST(Core_TExpr, assignment)
{
Mat a(18, 22, CV_32F), b(18, 22, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
Mat got = expr1("t = {0} * 2.5; t + {1}", { a, b });
Mat exp; cv::addWeighted(a, 2.5, b, 1.0, 0.0, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3);
}
// Tuple -> several outputs.
TEST(Core_TExpr, tuple_outputs)
{
Mat a(14, 19, CV_32F), b(14, 19, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
std::vector<Mat> out;
cv::texpr("({0} + {1}, {0} - {1})", std::vector<Mat>{ a, b }, out);
ASSERT_EQ(out.size(), 2u);
Mat eadd, esub; cv::add(a, b, eadd); cv::subtract(a, b, esub);
EXPECT_LE(cvtest::norm(out[0], eadd, NORM_INF), 1e-3) << "sum";
EXPECT_LE(cvtest::norm(out[1], esub, NORM_INF), 1e-3) << "diff";
}
// Grouping parens (NOT a tuple) inside a larger expression.
TEST(Core_TExpr, grouping_parens)
{
Mat a(11, 13, CV_32F), b(11, 13, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
Mat got = expr1("({0} + {1}) * 2", { a, b });
Mat exp; cv::add(a, b, exp); exp *= 2.0;
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3);
}
// Type-cast function: float -> uint8 (saturating).
TEST(Core_TExpr, cast_uint8)
{
Mat a(23, 17, CV_32F);
theRNG().fill(a, RNG::UNIFORM, -50.f, 300.f);
Mat got = expr1("uint8({0})", { a });
Mat exp; a.convertTo(exp, CV_8U);
ASSERT_EQ(got.type(), exp.type());
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF));
}
// pow() function call with a scalar exponent.
TEST(Core_TExpr, pow_call)
{
Mat a(16, 16, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 5.f);
Mat got = expr1("pow({0}, 2)", { a });
Mat exp; cv::pow(a, 2.0, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3);
}
// pow over the interesting exponents: the special-cased 2/3/0.5/1/0, the general exp/log path
// (2.5, -1.5), and negative bases (integer exponent -> exact signed result, fractional -> NaN),
// against the double std::pow reference. Sizes chosen to exercise both the SIMD body and the tail.
TEST(Core_TExpr, pow_exponents)
{
for (int depth : { CV_32F, CV_64F })
{
const double eps = depth == CV_32F ? 1e-6 : 1e-9;
Mat a0(37, 41, CV_64F), a;
theRNG().fill(a0, RNG::UNIFORM, 0.05, 9.);
a0.convertTo(a, depth);
for (double p : { 2., 3., 0.5, 1., 0., 2.5, -1.5, -0.5, 5., -2., 11. })
{
Mat got = expr1(cv::format("pow({0}, %.10g)", p), { a });
ASSERT_EQ(got.depth(), depth) << "p=" << p;
Mat ad, gd;
a.convertTo(ad, CV_64F); got.convertTo(gd, CV_64F);
double maxerr = 0;
for (int y = 0; y < a.rows; y++)
for (int x = 0; x < a.cols; x++)
{
double r = std::pow(ad.at<double>(y, x), p);
maxerr = std::max(maxerr, std::abs(gd.at<double>(y, x) - r) / std::max(1.0, std::abs(r)));
}
EXPECT_LE(maxerr, eps) << "depth=" << depth << " p=" << p;
}
}
}
// negative bases: integer exponents keep exact signed results (scalar patch path), a fractional
// exponent yields NaN - both matching std::pow
TEST(Core_TExpr, pow_negative_base)
{
Mat a(9, 13, CV_32F);
theRNG().fill(a, RNG::UNIFORM, -5.f, -1.f);
Mat got3 = expr1("pow({0}, 3)", { a });
for (int y = 0; y < a.rows; y++)
for (int x = 0; x < a.cols; x++)
ASSERT_NEAR(got3.at<float>(y, x), std::pow((double)a.at<float>(y, x), 3.), 1e-2);
Mat gotf = expr1("pow({0}, 2.5)", { a });
for (int y = 0; y < a.rows; y++)
for (int x = 0; x < a.cols; x++)
ASSERT_TRUE(cvIsNaN(gotf.at<float>(y, x))) << "pow(neg, frac) must be NaN";
}
// unary minus and abs() - compositions over the binary family (no dedicated kernels)
TEST(Core_TExpr, neg_abs)
{
for (int depth : { CV_32F, CV_16S })
{
Mat a(15, 19, depth);
theRNG().fill(a, RNG::UNIFORM, -100, 100);
Mat gneg = expr1("-{0}", { a });
Mat eneg; cv::subtract(Scalar(0), a, eneg);
ASSERT_EQ(gneg.depth(), depth);
EXPECT_EQ(0, cvtest::norm(gneg, eneg, NORM_INF)) << "neg depth=" << depth;
// abs == absdiff(a, 0) INCLUDING the texpr auto result type rule: signed input -> the
// UNSIGNED type of the same width (|SHRT_MIN| fits u16 exactly, no saturation). NB this
// deliberately differs from the public cv::absdiff, whose auto depth keeps the source
// type for 4.x compatibility - the VALUES agree, the depth rule is the engine's own.
Mat gabs = expr1("abs({0})", { a });
ASSERT_EQ(gabs.depth(), cv::ew::absdiffResultDepth(depth)) << "texpr absdiff type rule";
Mat eabs; cv::absdiff(a, Scalar(0), eabs);
eabs.convertTo(eabs, gabs.depth());
EXPECT_EQ(0, cvtest::norm(gabs, eabs, NORM_INF)) << "abs depth=" << depth;
}
}
// the abs(a - b) -> absdiff(a, b) peephole: on unsigned data the literal semantics (saturating
// subtract) would give max(a-b, 0) - the rewrite must give the true |a - b| everywhere
TEST(Core_TExpr, abs_sub_peephole)
{
for (int depth : { CV_8U, CV_16S, CV_32F })
{
Mat a(23, 17, depth), b(23, 17, depth);
theRNG().fill(a, RNG::UNIFORM, 0, 100);
theRNG().fill(b, RNG::UNIFORM, 0, 100);
Mat got = expr1("abs({0} - {1})", { a, b });
ASSERT_EQ(got.depth(), cv::ew::absdiffResultDepth(depth)) << "depth=" << depth;
Mat exp; cv::absdiff(a, b, exp);
exp.convertTo(exp, got.depth()); // cv::absdiff auto KEEPS the source depth (4.x)
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF)) << "depth=" << depth;
}
}
// clamp: scalar bounds (the common shape), array bounds, and type preservation
TEST(Core_TExpr, clamp)
{
for (int depth : { CV_8U, CV_16S, CV_32F, CV_64F })
{
Mat a(25, 31, depth);
theRNG().fill(a, RNG::UNIFORM, -100, 355);
Mat got = expr1("clamp({0}, 10, 200)", { a });
Mat emax, exp;
cv::max(a, 10.0, emax); cv::min(emax, 200.0, exp);
ASSERT_EQ(got.depth(), depth) << "clamp must keep the operand type";
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF)) << "depth=" << depth;
}
// array bounds
Mat x(14, 22, CV_32F), lo(14, 22, CV_32F), hi(14, 22, CV_32F);
theRNG().fill(x, RNG::UNIFORM, -10.f, 10.f);
theRNG().fill(lo, RNG::UNIFORM, -5.f, 0.f);
theRNG().fill(hi, RNG::UNIFORM, 0.f, 5.f);
Mat got = expr1("clamp({0}, {1}, {2})", { x, lo, hi });
Mat emax, exp;
cv::max(x, lo, emax); cv::min(emax, hi, exp);
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF));
}
// '**' operator: pow alias, binds tighter than '*', right-associative
TEST(Core_TExpr, pow_operator)
{
Mat a(13, 18, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 0.5f, 2.f);
Mat got = expr1("{0} ** 2", { a });
Mat exp; cv::pow(a, 2.0, exp);
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF));
// precedence: 3 * a ** 2 == 3 * (a ** 2)
Mat got2 = expr1("3 * {0} ** 2", { a });
Mat exp2 = 3.0 * exp;
EXPECT_LE(cvtest::norm(got2, exp2, NORM_INF), 1e-4);
// right associativity: a ** 2 ** 3 == a ** (2 ** 3) == a ** 8
Mat got3 = expr1("{0} ** 2 ** 3", { a });
Mat exp3; cv::pow(a, 8.0, exp3);
EXPECT_LE(cvtest::norm(got3, exp3, NORM_INF), 1e-4);
}
// '?:' conditional: select alias with the lowest precedence; right-associative chains
TEST(Core_TExpr, ternary_operator)
{
Mat a(19, 23, CV_32F), b(19, 23, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 0.f, 100.f);
theRNG().fill(b, RNG::UNIFORM, 0.f, 100.f);
// max via ?: - the condition is a full comparison (lower precedence than '>')
Mat got = expr1("{0} > {1} ? {0} : {1}", { a, b });
Mat exp; cv::max(a, b, exp);
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF));
// arithmetic in every position without parentheses
Mat got2 = expr1("{0} - {1} > 10 ? {0} + 1 : {1} * 2", { a, b });
Mat mask = (a - b > 10), e1 = a + 1, e2 = b * 2, exp2 = e2.clone();
e1.copyTo(exp2, mask);
EXPECT_LE(cvtest::norm(got2, exp2, NORM_INF), 1e-4);
// right-associative chain: c1 ? x : c2 ? y : z
Mat got3 = expr1("{0} > 66 ? 1 : {0} > 33 ? 2 : 3", { a });
Mat exp3(a.size(), CV_32F);
for (int y = 0; y < a.rows; y++)
for (int x = 0; x < a.cols; x++)
{
float v = a.at<float>(y, x);
exp3.at<float>(y, x) = v > 66 ? 1.f : v > 33 ? 2.f : 3.f;
}
EXPECT_EQ(0, cvtest::norm(got3, exp3, NORM_INF));
}
// hypot(x, y) / its cv-flavored alias mag(x, y): naive sqrt(x^2 + y^2), matching cv::magnitude;
// kernels exist for the float depths only (f16/bf16/f32/f64)
TEST(Core_TExpr, hypot_mag)
{
for (int depth : { CV_32F, CV_64F })
{
Mat x(23, 31, depth), y(23, 31, depth);
theRNG().fill(x, RNG::UNIFORM, -100, 100);
theRNG().fill(y, RNG::UNIFORM, -100, 100);
Mat got = expr1("hypot({0}, {1})", { x, y });
ASSERT_EQ(got.depth(), depth);
Mat exp; cv::magnitude(x, y, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), depth == CV_32F ? 1e-4 : 1e-9);
Mat got2 = expr1("mag({0}, {1})", { x, y }); // alias
EXPECT_EQ(0, cvtest::norm(got2, got, NORM_INF));
}
// broadcast branch: hypot(array, scalar)
Mat x(11, 17, CV_32F);
theRNG().fill(x, RNG::UNIFORM, -10.f, 10.f);
Mat got = expr1("hypot({0}, 3)", { x });
for (int r = 0; r < x.rows; r++)
for (int c = 0; c < x.cols; c++)
{
float v = x.at<float>(r, c);
ASSERT_NEAR(got.at<float>(r, c), std::sqrt(v*v + 9.f), 1e-4) << r << "," << c;
}
// f16: T -> T through the native kernel (f32 hub inside)
Mat xh, yh, y16(11, 17, CV_32F);
x.convertTo(xh, CV_16F);
theRNG().fill(y16, RNG::UNIFORM, -10.f, 10.f);
y16.convertTo(yh, CV_16F);
Mat goth = expr1("hypot({0}, {1})", { xh, yh });
ASSERT_EQ(goth.depth(), CV_16F);
}
// atan2(y, x): radians, the standard C range (-pi, pi], all four quadrants; the f32 kernel is the
// fastAtan2 minimax polynomial (~1e-5 rad absolute), f64 is exact std::atan2
TEST(Core_TExpr, atan2)
{
Mat y(23, 31, CV_32F), x(23, 31, CV_32F);
theRNG().fill(y, RNG::UNIFORM, -10.f, 10.f); // both signs -> all quadrants
theRNG().fill(x, RNG::UNIFORM, -10.f, 10.f);
Mat got = expr1("atan2({0}, {1})", { y, x });
ASSERT_EQ(got.depth(), CV_32F);
double maxerr = 0;
for (int r = 0; r < y.rows; r++)
for (int c = 0; c < y.cols; c++)
{
double ref = std::atan2((double)y.at<float>(r, c), (double)x.at<float>(r, c));
maxerr = std::max(maxerr, std::abs((double)got.at<float>(r, c) - ref));
}
EXPECT_LE(maxerr, 2e-4) << "fastAtan2-class polynomial accuracy (measured ~1.6e-4 rad)";
// f64: exact std::atan2 per element
Mat y64, x64;
y.convertTo(y64, CV_64F); x.convertTo(x64, CV_64F);
Mat got64 = expr1("atan2({0}, {1})", { y64, x64 });
ASSERT_EQ(got64.depth(), CV_64F);
for (int r = 0; r < y.rows; r++)
for (int c = 0; c < y.cols; c++)
ASSERT_EQ(got64.at<double>(r, c),
std::atan2(y64.at<double>(r, c), x64.at<double>(r, c)));
// axis cases: atan2(0, 1) = 0, atan2(1, 0) = pi/2, atan2(0, -1) = pi, atan2(-1, 0) = -pi/2
Mat ya = (Mat_<float>(1, 4) << 0.f, 1.f, 0.f, -1.f);
Mat xa = (Mat_<float>(1, 4) << 1.f, 0.f, -1.f, 0.f);
Mat ga = expr1("atan2({0}, {1})", { ya, xa });
const float expctd[] = { 0.f, (float)(CV_PI/2), (float)CV_PI, (float)(-CV_PI/2) };
for (int i = 0; i < 4; i++)
ASSERT_NEAR(ga.at<float>(0, i), expctd[i], 1e-4) << "axis case " << i;
}
// per-element (array) exponent
TEST(Core_TExpr, pow_array_exponent)
{
Mat a(21, 27, CV_32F), b(21, 27, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 0.1f, 5.f);
theRNG().fill(b, RNG::UNIFORM, -2.f, 3.f);
Mat got = expr1("pow({0}, {1})", { a, b });
double maxerr = 0;
for (int y = 0; y < a.rows; y++)
for (int x = 0; x < a.cols; x++)
{
double r = std::pow((double)a.at<float>(y, x), (double)b.at<float>(y, x));
maxerr = std::max(maxerr, std::abs(got.at<float>(y, x) - r) / std::max(1.0, std::abs(r)));
}
EXPECT_LE(maxerr, 1e-6);
}
// ---------------------------------------------------------------------------------- unary math
// Golden result = the double-precision std:: function applied per element (never the op under
// test); the tolerance is relative, scaled by the output depth's precision.
typedef double (*mathRef)(double);
struct MathOpRef { const char* name; mathRef ref; };
static const MathOpRef mathOps[] = {
{ "sqrt", std::sqrt }, { "exp", std::exp }, { "log", std::log },
{ "sin", std::sin }, { "cos", std::cos }, { "tanh", std::tanh }, { "erf", std::erf },
{ "relu", [](double x) { return x > 0 ? x : 0.; } },
};
// max |got - ref| / max(1, |ref|) over the array, both evaluated in f64
static double relErr(const Mat& got, const Mat& in, mathRef ref)
{
Mat gotd, ind;
got.convertTo(gotd, CV_64F);
in.convertTo(ind, CV_64F);
double maxerr = 0;
for (int y = 0; y < ind.rows; y++)
for (int x = 0; x < ind.cols; x++)
{
double r = ref(ind.at<double>(y, x));
double e = std::abs(gotd.at<double>(y, x) - r) / std::max(1.0, std::abs(r));
maxerr = std::max(maxerr, e);
}
return maxerr;
}
typedef testing::TestWithParam<MatDepth> Core_TExpr_Math;
TEST_P(Core_TExpr_Math, unary_accuracy)
{
const int depth = GetParam();
// eps: half-precision types are exact to ~2^-8/2^-11 per element; the f32 kernels are
// polynomial approximations (a few ulp); f64 sqrt/exp/log are also vectorized polynomials
const double eps = depth == CV_16F ? 2e-3 : depth == CV_16BF ? 1.6e-2
: depth == CV_32F ? 1e-6 : 1e-9;
Mat a0(37, 41, CV_32F);
theRNG().fill(a0, RNG::UNIFORM, 0.05f, 9.f); // positive: one range serves log/sqrt too
Mat a;
a0.convertTo(a, depth);
for (const MathOpRef& m : mathOps)
{
Mat got = expr1(cv::format("%s({0})", m.name), { a });
ASSERT_EQ(got.depth(), depth) << m.name << ": math must be T -> T on float inputs";
EXPECT_LE(relErr(got, a, m.ref), eps) << m.name << " depth=" << depth;
}
// negative inputs for the ops defined there (skip log/sqrt)
Mat b0(37, 41, CV_32F), b;
theRNG().fill(b0, RNG::UNIFORM, -8.f, 8.f);
b0.convertTo(b, depth);
for (const char* name : { "exp", "sin", "cos", "tanh", "erf", "relu" })
{
const MathOpRef* m = nullptr;
for (const MathOpRef& c : mathOps) if (!strcmp(c.name, name)) m = &c;
Mat got = expr1(cv::format("%s({0})", name), { b });
EXPECT_LE(relErr(got, b, m->ref), eps) << name << "(neg) depth=" << depth;
}
}
INSTANTIATE_TEST_CASE_P(/**/, Core_TExpr_Math,
testing::Values(CV_16F, CV_16BF, CV_32F, CV_64F));
// integer input computes in the float domain and lands in f32
TEST(Core_TExpr, math_int_input)
{
Mat a(19, 23, CV_8U);
theRNG().fill(a, RNG::UNIFORM, 1, 100);
Mat got = expr1("sqrt({0})", { a });
ASSERT_EQ(got.depth(), CV_32F);
EXPECT_LE(relErr(got, a, std::sqrt), 1e-6);
}
// in-place unary math (dst aliases src): the tail backoff must not re-apply the op
TEST(Core_TExpr, math_inplace)
{
Mat a(21, 31, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 0.1f, 9.f);
Mat ref = expr1("sqrt({0})", { a });
std::vector<Mat> out{ a }; // preallocated == input => in-place
cv::texpr("sqrt({0})", std::vector<Mat>{ a }, out);
EXPECT_EQ(0, cvtest::norm(out[0], ref, NORM_INF));
}
// ------------------------------------------------------------------------------------- select
TEST(Core_TExpr, select_basic)
{
for (int type : { CV_8UC1, CV_16SC1, CV_32FC1, CV_64FC1 })
{
Mat a(25, 33, type), b(25, 33, type);
theRNG().fill(a, RNG::UNIFORM, 0, 100);
theRNG().fill(b, RNG::UNIFORM, 0, 100);
Mat got = expr1("select({0} > {1}, {0}, {1})", { a, b }); // == max(a, b)
Mat exp; cv::max(a, b, exp);
ASSERT_EQ(got.type(), exp.type()) << "type=" << type;
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF)) << "type=" << type;
}
}
// one branch is a scalar constant (broadcast stepx == 0 inside the kernel)
TEST(Core_TExpr, select_const_branch)
{
Mat a(17, 29, CV_32F);
theRNG().fill(a, RNG::UNIFORM, -10.f, 10.f);
Mat got = expr1("select({0} > 0, {0}, 0)", { a }); // == relu
Mat exp = expr1("relu({0})", { a });
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF));
}
// non-1-byte mask is normalized via `mask != 0`, not a value cast
TEST(Core_TExpr, select_float_mask)
{
Mat m(15, 27, CV_32F), a(15, 27, CV_32F), b(15, 27, CV_32F);
theRNG().fill(m, RNG::UNIFORM, -1.f, 1.f);
theRNG().fill(a, RNG::UNIFORM, 0.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 0.f, 10.f);
m.at<float>(3, 5) = 0.f; // exact zero -> must take branch b
m.at<float>(7, 7) = 0.25f; // would round/saturate to 0 under a value cast
Mat got = expr1("select({0}, {1}, {2})", { m, a, b });
Mat mask = (m != 0), exp;
exp = b.clone(); a.copyTo(exp, mask);
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF));
}
}} // namespace
modules/core/test/test_new_arithm.cpp
+364
View File
@@ -0,0 +1,364 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// White-box tests for the element-wise engine internals (cv::ew, declared in the module-internal
// src/arithm_expr.hpp - NOT part of the public API; the public surface is cv::add/... and cv::texpr,
// covered by test_new_arithm_extensive.cpp / test_arithm_expr.cpp). Two groups:
// - the type-inference + cast-insertion policy (emitBinary) compiled and run through the executor;
// - the single-op vertical slice (makeBinaryArithProgram / maybeAddCast) end-to-end.
// Both check against the classic cv:: ops.
#include "test_precomp.hpp"
#include "../src/arithm_expr.hpp"
namespace opencv_test { namespace {
using namespace cv::ew;
// emit shortcuts: a flexible literal, and a binary op over two slots.
static int K(TExpr& e, double v) { return e.addConst(EW_DEPTH_NONE, Scalar(v), 1); }
static int bin(TExpr& e, TOp op, int a, int b){ return e.emitBinary(op, a, b); }
// Compile `e` and run it over the given inputs. The operands were already typed at build time
// (addInput carries each input's depth), so compile() just binds kernels + packs temp buffers.
static std::vector<Mat> run(TExpr& e, const std::vector<Mat>& inps)
{
e.compile();
std::vector<Mat> outs(e.noutputs);
e.exec(inps.data(), outs.data());
return outs;
}
// addWeighted(a,alpha,b,beta,gamma) = a*alpha + b*beta + gamma, built op-by-op via emitBinary
// (the temp buffers are allocated automatically by compile()'s liveness pass).
TEST(Core_EW_Compile, addweighted_f32)
{
const int chans[] = { 1, 3 };
double alpha = 2.5, beta = -1.5, gamma = 7.0;
for (int ci = 0; ci < 2; ci++)
{
int H = 19, W = 23, cn = chans[ci];
Mat a(H, W, CV_32FC(cn)), b(H, W, CV_32FC(cn));
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
TExpr g;
int ia = g.addInput(CV_32F), ib = g.addInput(CV_32F);
int t0 = bin(g, OP_MUL, ia, K(g, alpha));
int t1 = bin(g, OP_MUL, ib, K(g, beta));
int t2 = bin(g, OP_ADD, t0, t1);
g.output(bin(g, OP_ADD, t2, K(g, gamma)));
std::vector<Mat> out = run(g, { a, b });
Mat exp; cv::addWeighted(a, alpha, b, beta, gamma, exp);
ASSERT_EQ(out[0].type(), exp.type());
EXPECT_LE(cvtest::norm(out[0], exp, NORM_INF), 1e-3) << "cn=" << cn;
}
}
// Mixed integer types: out = saturate_u8( saturate_u8(a*2.5) + b ), a,b are u8.
// emitBinary must insert u8->f32 input casts and f32->u8 result casts around each op (2.5 does not
// fit u8, so the direct u8 kernel is refused and the float working path is taken).
TEST(Core_EW_Compile, mixed_u8_inserts_casts)
{
int H = 16, W = 24;
Mat a(H, W, CV_8U), b(H, W, CV_8U);
theRNG().fill(a, RNG::UNIFORM, 0, 60);
theRNG().fill(b, RNG::UNIFORM, 0, 60);
TExpr g;
int ia = g.addInput(CV_8U), ib = g.addInput(CV_8U);
int mul = bin(g, OP_MUL, ia, K(g, 2.5)); // -> u8 (natural)
g.output(bin(g, OP_ADD, mul, ib)); // -> u8
std::vector<Mat> out = run(g, { a, b });
Mat t0, exp;
a.convertTo(t0, CV_8U, 2.5); // saturate_u8(a*2.5)
cv::add(t0, b, exp); // saturate_u8(t0 + b)
ASSERT_EQ(out[0].type(), exp.type());
EXPECT_EQ(0, cvtest::norm(out[0], exp, NORM_INF));
}
// Tuple of two outputs from shared inputs: (a+b, a-b).
TEST(Core_EW_Compile, multi_output_tuple)
{
int H = 14, W = 18;
Mat a(H, W, CV_32F), b(H, W, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
TExpr g;
int ia = g.addInput(CV_32F), ib = g.addInput(CV_32F);
g.output(bin(g, OP_ADD, ia, ib));
g.output(bin(g, OP_SUB, ia, ib));
std::vector<Mat> out = run(g, { a, b });
ASSERT_EQ(out.size(), 2u);
Mat eadd, esub; cv::add(a, b, eadd); cv::subtract(a, b, esub);
EXPECT_LE(cvtest::norm(out[0], eadd, NORM_INF), 1e-3) << "sum";
EXPECT_LE(cvtest::norm(out[1], esub, NORM_INF), 1e-3) << "diff";
}
// Liveness: a linear chain of temps with disjoint lifetimes must share physical buffers.
// out = (((a+1)+1)+1)+1 == a+4 : the last add is redirected straight into the output slot, the
// three live temps share just 2 physical buffers.
TEST(Core_EW_Compile, temp_buffer_reuse)
{
int H = 10, W = 13;
Mat a(H, W, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
TExpr g;
int x = g.addInput(CV_32F);
for (int k = 0; k < 4; k++)
x = bin(g, OP_ADD, x, K(g, 1.0));
g.output(x);
std::vector<Mat> out = run(g, { a });
// last instruction writes straight into the OUTPUT slot (its producing temp was redirected)
EXPECT_EQ(g.arginfo[g.prog[g.prog.size() - 1].result].kind, TExpr::OUTPUT);
EXPECT_EQ(g.nbuffers, 2); // disjoint lifetimes => only 2 physical buffers
Mat exp; cv::add(a, Scalar(4.0), exp);
EXPECT_LE(cvtest::norm(out[0], exp, NORM_INF), 1e-4);
}
// promoteArith is the auto result-depth rule (rdepth == -1). Checked against an INDEPENDENT hardcoded
// table (NOT computed from the engine): the extensive tests feed promoteArith to BOTH the engine and
// their own reference, so a wrong-but-consistent rule slips through there - this catches it. Also
// asserts commutativity, which a max-rank scheme silently breaks for mixed-sign / same-width floats.
TEST(Core_EW_Compile, promoteArith_rules)
{
struct { int a, b, want; } cases[] = {
// same signedness -> the wider one, sign kept
{ CV_8U, CV_8U, CV_8U }, { CV_8U, CV_16U, CV_16U }, { CV_8U, CV_64U, CV_64U },
{ CV_16S, CV_64S, CV_64S }, { CV_8S, CV_32S, CV_32S },
// mixed sign, same width -> next-wider signed (64-bit has no wider int -> f64)
{ CV_8U, CV_8S, CV_16S }, { CV_16U, CV_16S, CV_32S },
{ CV_32U, CV_32S, CV_64S }, { CV_64U, CV_64S, CV_64F },
// mixed sign, different width
{ CV_8S, CV_16U, CV_32S }, { CV_8U, CV_16S, CV_16S }, { CV_32S, CV_64U, CV_64F },
// float + int -> smallest covering float
{ CV_16F, CV_8U, CV_16F }, { CV_16BF, CV_8U, CV_16BF }, { CV_16F, CV_16U, CV_32F },
{ CV_16F, CV_32S, CV_64F }, { CV_32F, CV_32S, CV_64F }, { CV_32F, CV_16S, CV_32F },
// float + float
{ CV_16F, CV_32F, CV_32F }, { CV_16F, CV_16BF, CV_32F }, { CV_64F, CV_8U, CV_64F },
// flexible operand (EW_DEPTH_NONE) does not force promotion
{ EW_DEPTH_NONE, CV_16U, CV_16U }, { CV_16U, EW_DEPTH_NONE, CV_16U },
{ EW_DEPTH_NONE, EW_DEPTH_NONE, EW_DEPTH_NONE },
};
for (auto& c : cases)
{
int got = promoteArith(c.a, c.b);
EXPECT_EQ(got, c.want) << "promoteArith(" << c.a << "," << c.b << ")";
EXPECT_EQ(promoteArith(c.b, c.a), got) << "not commutative at " << c.a << "," << c.b;
}
// exhaustive commutativity over all real depths
const int depths[] = { CV_8U, CV_8S, CV_16U, CV_16S, CV_32U, CV_32S, CV_64U, CV_64S,
CV_16F, CV_16BF, CV_32F, CV_64F };
for (int a : depths) for (int b : depths)
EXPECT_EQ(promoteArith(a, b), promoteArith(b, a)) << "noncommutative at " << a << "," << b;
}
// ---------------------------------------------------------------------------
// Vertical slice: single-op programs (ADD/SUB/MUL/DIV/POW f32 and CAST) built via the hand builders
// (makeBinaryArithProgram / maybeAddCast) and run through the executor, checked against classic cv::.
// ---------------------------------------------------------------------------
// out = op(a, b), composed via the general binary-arith builder (the engine backing cv::add).
static Mat runBinary(TOp op, const Mat& a, const Mat& b, int rdepth)
{
TExpr p; makeBinaryArithProgram(p, op, a.depth(), b.depth(), rdepth);
Mat inps[] = {a, b}, out;
p.exec(inps, &out);
return out;
}
// out = cast(a), built through maybeAddCast (a single OP_CAST) and compiled.
static Mat runCast(const Mat& a, int rdepth)
{
TExpr e;
int s = e.addInput(a.depth());
e.output(e.maybeAddCast(s, rdepth));
e.compile();
Mat out;
e.exec(&a, &out);
return out;
}
static void cvRef(TOp op, const Mat& a, const Mat& b, Mat& dst)
{
switch (op)
{
case OP_ADD: cv::add(a, b, dst); break;
case OP_SUB: cv::subtract(a, b, dst); break;
case OP_MUL: cv::multiply(a, b, dst); break;
case OP_DIV: cv::divide(a, b, dst); break;
default: CV_Error(Error::StsBadArg, "unexpected op");
}
}
// ADD/SUB/MUL/DIV on f32, single- and multi-channel, same shape.
TEST(Core_EW_Slice, binary_f32_same_shape)
{
const TOp ops[] = { OP_ADD, OP_SUB, OP_MUL, OP_DIV };
const int chans[] = { 1, 3, 4 };
RNG& rng = theRNG();
for (int oi = 0; oi < 4; oi++)
for (int ci = 0; ci < 3; ci++)
{
int H = 17, W = 33, cn = chans[ci];
Mat a(H, W, CV_32FC(cn)), b(H, W, CV_32FC(cn));
rng.fill(a, RNG::UNIFORM, 1.f, 10.f);
rng.fill(b, RNG::UNIFORM, 1.f, 10.f);
Mat got = runBinary(ops[oi], a, b, CV_32F);
Mat exp; cvRef(ops[oi], a, b, exp);
ASSERT_EQ(got.size(), exp.size());
ASSERT_EQ(got.type(), exp.type());
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3)
<< "op=" << opName(ops[oi]) << " cn=" << cn;
}
}
// POW with a 1x1 (broadcast) exponent vs cv::pow(a, scalar).
TEST(Core_EW_Slice, pow_f32_scalar_exp)
{
Mat a(20, 25, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 5.f);
Mat e(1, 1, CV_32F, Scalar(2.0));
Mat got = runBinary(OP_POW, a, e, CV_32F);
Mat exp; cv::pow(a, 2.0, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3);
}
// Row / column broadcasting (single channel) checked against repeat()+cv::add.
TEST(Core_EW_Slice, broadcast_row_col)
{
int H = 12, W = 19;
Mat a(H, W, CV_32F);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
{ // row vector broadcast over rows
Mat brow(1, W, CV_32F);
theRNG().fill(brow, RNG::UNIFORM, 1.f, 10.f);
Mat got = runBinary(OP_ADD, a, brow, CV_32F);
Mat bb, exp; cv::repeat(brow, H, 1, bb); cv::add(a, bb, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3) << "row";
}
{ // column vector broadcast over columns
Mat bcol(H, 1, CV_32F);
theRNG().fill(bcol, RNG::UNIFORM, 1.f, 10.f);
Mat got = runBinary(OP_ADD, a, bcol, CV_32F);
Mat bb, exp; cv::repeat(bcol, 1, W, bb); cv::add(a, bb, exp);
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3) << "col";
}
}
// Channel broadcasting: HxWx3 * HxWx1 -> HxWx3.
TEST(Core_EW_Slice, broadcast_channel)
{
int H = 15, W = 21;
Mat a(H, W, CV_32FC3), b(H, W, CV_32FC1);
theRNG().fill(a, RNG::UNIFORM, 1.f, 10.f);
theRNG().fill(b, RNG::UNIFORM, 1.f, 10.f);
Mat got = runBinary(OP_MUL, a, b, CV_32F);
std::vector<Mat> ach; cv::split(a, ach);
for (size_t c = 0; c < ach.size(); c++) cv::multiply(ach[c], b, ach[c]);
Mat exp; cv::merge(ach, exp);
ASSERT_EQ(got.type(), exp.type());
EXPECT_LE(cvtest::norm(got, exp, NORM_INF), 1e-3);
}
// Saturating cast f32 <-> {u8, s32}.
TEST(Core_EW_Slice, cast_basic)
{
Mat a(23, 31, CV_32F);
theRNG().fill(a, RNG::UNIFORM, -50.f, 300.f); // exercise saturation for u8
{
Mat got = runCast(a, CV_8U);
Mat exp; a.convertTo(exp, CV_8U);
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF)) << "f32->u8";
}
{
Mat got = runCast(a, CV_32S);
Mat exp; a.convertTo(exp, CV_32S);
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF)) << "f32->s32";
}
{
Mat u; a.convertTo(u, CV_8U);
Mat got = runCast(u, CV_32F);
Mat exp; u.convertTo(exp, CV_32F);
EXPECT_EQ(0, cvtest::norm(got, exp, NORM_INF)) << "u8->f32";
}
}
// cv::add / cv::subtract on 32-bit ints SATURATE (SIMD via the local v_add_sat/v_sub_sat, scalar
// tail via the int64 work type) - uniform random data essentially never crosses the boundaries,
// so directed cases are mandatory (see the v_sat_arith brief): both rails, the 0 - INT_MIN case,
// mixed-sign non-overflow, and a random block checked against an exact int64 reference.
TEST(Core_EW_AddSub, saturation_s32_u32)
{
const int W = 37; // odd width: SIMD body + scalar tail both covered
{
const int mx = INT_MAX, mn = INT_MIN;
const int a[] = { mx, mn, mx, mn, 0, -1, mx, 12345 };
const int b[] = { 1, -1, mx, mn, mn, mx, -1, -54321 };
// exact int64 references
Mat A(1, 8, CV_32S, (void*)a), B(1, 8, CV_32S, (void*)b), sum, dif;
cv::add(A, B, sum);
cv::subtract(A, B, dif);
for (int i = 0; i < 8; i++)
{
int64_t rs = (int64_t)a[i] + b[i], rd = (int64_t)a[i] - b[i];
EXPECT_EQ(sum.at<int>(i), (int)std::min<int64_t>(std::max<int64_t>(rs, mn), mx)) << "add s32 case " << i;
EXPECT_EQ(dif.at<int>(i), (int)std::min<int64_t>(std::max<int64_t>(rd, mn), mx)) << "sub s32 case " << i;
}
}
{
const unsigned mx = UINT_MAX;
const unsigned a[] = { mx, 0, mx, 5, 0, 100 };
const unsigned b[] = { 1, 1, mx, 5, 0, 7 };
Mat A(1, 6, CV_32U, (void*)a), B(1, 6, CV_32U, (void*)b), sum, dif;
cv::add(A, B, sum);
cv::subtract(A, B, dif);
for (int i = 0; i < 6; i++)
{
uint64_t rs = (uint64_t)a[i] + b[i];
int64_t rd = (int64_t)a[i] - b[i];
EXPECT_EQ(sum.at<unsigned>(i), (unsigned)std::min<uint64_t>(rs, mx)) << "add u32 case " << i;
EXPECT_EQ(dif.at<unsigned>(i), (unsigned)std::max<int64_t>(rd, 0)) << "sub u32 case " << i;
}
}
// random block spanning the full range (so saturation DOES occur), vs the int64 reference
{
Mat a(15, W, CV_32S), b(15, W, CV_32S), sum, dif;
theRNG().fill(a, RNG::UNIFORM, INT_MIN, INT_MAX);
theRNG().fill(b, RNG::UNIFORM, INT_MIN, INT_MAX);
cv::add(a, b, sum);
cv::subtract(a, b, dif);
for (int y = 0; y < a.rows; y++)
for (int x = 0; x < W; x++)
{
int64_t rs = (int64_t)a.at<int>(y, x) + b.at<int>(y, x);
int64_t rd = (int64_t)a.at<int>(y, x) - b.at<int>(y, x);
ASSERT_EQ(sum.at<int>(y, x), (int)std::min<int64_t>(std::max<int64_t>(rs, INT_MIN), INT_MAX)) << y << "," << x;
ASSERT_EQ(dif.at<int>(y, x), (int)std::min<int64_t>(std::max<int64_t>(rd, INT_MIN), INT_MAX)) << y << "," << x;
}
}
}
}} // namespace
modules/core/test/test_new_arithm_extensive.cpp
+551
View File
@@ -0,0 +1,551 @@
// This file is part of OpenCV project.
// It is subject to the license terms in the LICENSE file found in the top-level directory
// of this distribution and at http://opencv.org/license.html.
// Randomized property-based accuracy tests for the broadcasting element-wise ops, exercised through
// the PUBLIC cv:: entry points (add/subtract/multiply/divide/min/max/absdiff/compare with a mask,
// dtype, in-place aliasing and mixed input types). For each caseidx a deterministic splitmix64 seed
// picks random depths and broadcast-compatible shapes (ndims<=4, total<=100000) and every axis of each
// operand independently keeps its size or drops to 1, so full/row/col/channel broadcast are all hit.
// The reference decomposes each op into per-channel cv::broadcast + convertTo + the same op on aligned
// same-shape single-channel arrays. The module-internal header is included only for the promotion rule
// (promoteArith / absdiffResultDepth) the reference needs to predict each op's auto result depth.
#include "test_precomp.hpp"
#include "../src/arithm_expr.hpp"
namespace opencv_test { namespace {
using namespace cv::ew;
static inline uint64_t mix64(uint64_t x)
{
x += 0x9E3779B97F4A7C15ULL;
x = (x ^ (x >> 30)) * 0xBF58476D1CE4E5B9ULL;
x = (x ^ (x >> 27)) * 0x94D049BB133111EBULL;
return x ^ (x >> 31);
}
static const uint64_t kSuiteSalt = 0x9ADD0CA57ULL;
static const int kNumCases = 1000;
static const int kMaxElems = 100000;
// engine-supported depths
static const int kDepths[] = { CV_8U, CV_8S, CV_16U, CV_16S, CV_32U, CV_32S,
CV_64U, CV_64S, CV_16F, CV_16BF, CV_32F, CV_64F };
static int sampleDepth(RNG& rng) { return kDepths[rng.uniform(0, (int)(sizeof(kDepths)/sizeof(kDepths[0])))]; }
static bool isFloat(int d) { return d==CV_16F || d==CV_16BF || d==CV_32F || d==CV_64F; }
// numpy-ish promotion (must mirror ew_exec.cpp's promote2 so the reference adds the same values)
static int promote2(int a, int b)
{
if (a == b) return a;
constexpr unsigned lbits = 3, lmask = (1u << lbits) - 1u;
const uint64_t typelut = (uint64_t)((0ULL << CV_8U*lbits) | (0ULL << CV_8S*lbits) |
(1ULL << CV_16U*lbits) | (1ULL << CV_16S*lbits) |
(2ULL << CV_32U*lbits) | (2ULL << CV_32S*lbits) |
(3ULL << CV_16F*lbits) | (3ULL << CV_16BF*lbits) |
(3ULL << CV_32F*lbits) | (4ULL << CV_64F*lbits) |
(4ULL << CV_64S*lbits) | (4ULL << CV_64U*lbits));
unsigned pr_a = unsigned((typelut >> (a*lbits)) & lmask);
unsigned pr_b = unsigned((typelut >> (b*lbits)) & lmask);
unsigned max_pr = std::max(pr_a, pr_b);
constexpr unsigned dbits = CV_CN_SHIFT, dmask = (1u << dbits) - 1u;
const unsigned ctypelut = ((CV_16S << 0*dbits) | (CV_32S << 1*dbits) | (CV_64S << 2*dbits) |
(CV_32F << 3*dbits) | (CV_64F << 4*dbits));
return int((ctypelut >> (max_pr*dbits)) & dmask);
}
// data range per depth: 8/16-bit wide enough to exercise saturation; 32/64-bit kept modest.
static void depthRange(int d, double& lo, double& hi)
{
switch (d)
{
case CV_8U: lo = 0; hi = 255; break;
case CV_8S: lo = -128; hi = 127; break;
case CV_16U: lo = 0; hi = 65535; break;
case CV_16S: lo = -32768; hi = 32767; break;
default: lo = -1000; hi = 1000; break; // 32/64-bit ints and floats: no overflow
}
}
static int sampleSize(RNG& rng, int lo, int hi)
{
if (hi <= lo) return lo;
if (rng.uniform(0.0, 1.0) < 0.35)
{
static const int cand[] = {1,2,3,4,7,8,15,16,17,31,32,33};
int picks[16], n = 0;
for (int c : cand) if (c >= lo && c <= hi) picks[n++] = c;
picks[n++] = lo; picks[n++] = hi;
return picks[rng.uniform(0, n)];
}
double v = std::exp(rng.uniform(std::log((double)lo), std::log((double)hi)));
return std::min(hi, std::max(lo, cvRound(v)));
}
// a random shape (ndims in 1..4) with product <= kMaxElems
static std::vector<int> sampleShape(RNG& rng)
{
int nd = rng.uniform(1, 5);
std::vector<int> s(nd);
long long prod = 1;
for (int d = 0; d < nd; d++)
{
int hi = (int)std::min<long long>(512, std::max<long long>(1, kMaxElems / prod));
s[d] = sampleSize(rng, 1, hi);
prod *= s[d];
}
return s;
}
// build a random Mat of the given depth & shape, filled via a CV_64F master (randUni can't fill
// 16f/16bf/32u/64u/64s directly), values in the per-depth range. ~1/3 of the time the result is
// a NON-contiguous sub-array: the parent is padded by 1..2 on each edge of each axis and we
// return the inner view (gapped outer steps), to exercise the engine's non-continuous path.
static Mat makeRandom(RNG& rng, const std::vector<int>& shape, int cn, int depth,
double rlo = 1, double rhi = 0)
{
const bool crop = rng.uniform(0, 3) == 0;
const int nd = (int)shape.size();
std::vector<int> pad(nd);
std::vector<Range> ranges(nd);
for (int d = 0; d < nd; d++)
{
int lo = crop ? rng.uniform(1, 3) : 0; // 1..2
int hi = crop ? rng.uniform(1, 3) : 0;
pad[d] = shape[d] + lo + hi;
ranges[d] = Range(lo, lo + shape[d]);
}
Mat m64(nd, pad.data(), CV_MAKETYPE(CV_64F, cn));
double lo = rlo, hi = rhi;
if (rlo > rhi) depthRange(depth, lo, hi); // rlo>rhi (default) => per-depth range
cvtest::randUni(rng, m64, Scalar::all(lo), Scalar::all(hi));
Mat big; m64.convertTo(big, CV_MAKETYPE(depth, cn));
return big(ranges); // full range when !crop => contiguous
}
static std::string shapeStr(const std::vector<int>& s)
{
std::string r = "[";
for (size_t i = 0; i < s.size(); i++) r += (i ? "x" : "") + std::to_string(s[i]);
return r + "]";
}
// per-output-depth tolerance. `floatPath` = a float was involved on the way to an integer
// output, so a final float->int rounding tie may differ from cv:: by 1 (benign).
static void checkClose(const Mat& got, const Mat& ref, int rdepth, bool floatPath, const char* what)
{
ASSERT_EQ(got.dims, ref.dims) << what;
ASSERT_EQ(got.type(), ref.type()) << what;
double n = cvtest::norm(got, ref, NORM_INF);
if (!isFloat(rdepth))
{
EXPECT_LE(n, floatPath ? 1.0 : 0.0) << what; // integer output: exact, or ±1 via float
}
else
{
double scale = std::max(1.0, cvtest::norm(ref, NORM_INF));
double rel = rdepth==CV_16BF ? 1e-2 : rdepth==CV_16F ? 2e-3 : 1e-5;
EXPECT_LE(n, rel*scale) << what << " (n=" << n << " scale=" << scale << ")";
}
}
// ------------------------------------------------------------------------------- add / sub
// Parameterized on (op, caseidx): op 0 = ADD, 1 = SUB. The two ops share the same per-case data
// (seed depends only on caseidx), so they run on identical inputs.
class EW_Extensive_BinOp : public ::testing::TestWithParam<std::tuple<int,int>> {};
TEST_P(EW_Extensive_BinOp, accuracy)
{
const int opSel = std::get<0>(GetParam());
const int caseidx = std::get<1>(GetParam());
const TOp op = opSel ? OP_SUB : OP_ADD;
const char* opStr = opSel ? "sub" : "add";
RNG rng(mix64(kSuiteSalt ^ (uint64_t)caseidx));
std::vector<int> shape = sampleShape(rng);
const int da = sampleDepth(rng), db = sampleDepth(rng);
// channels: pick a base count (biased toward 1); each operand keeps it or drops to 1, so we
// get C1+C1 (fold), Cn+Cn (fold) and the Cn+C1 / C1+Cn channel-broadcast (CH_DIM) mix.
static const int cncand[] = { 1, 1, 2, 3, 4 };
const int rcn = cncand[rng.uniform(0, 5)];
const int cn_a = rng.uniform(0, 2) ? rcn : 1;
const int cn_b = rng.uniform(0, 2) ? rcn : 1;
const int ocn = std::max(cn_a, cn_b);
// each operand independently keeps or broadcasts (->1) every axis
std::vector<int> sa(shape.size()), sb(shape.size()), res(shape.size());
for (size_t d = 0; d < shape.size(); d++)
{
sa[d] = rng.uniform(0, 2) ? shape[d] : 1;
sb[d] = rng.uniform(0, 2) ? shape[d] : 1;
res[d] = std::max(sa[d], sb[d]);
}
// in-place: ~1/3 of cases attempt it. When an input is spatially & channel "full" (its shape ==
// the output shape res, channels == ocn), alias the output onto it and force Tr to that input's
// depth so the buffer is truly reused. Exercises the executor's in-place handling AND the
// kernels' dst==src aliasing (the halide-tail backoff). Otherwise Tr is a free random depth.
int aliasIn = -1;
if (rng.uniform(0, 3) == 0)
{
if (sa == res && cn_a == ocn) aliasIn = 0;
else if (sb == res && cn_b == ocn) aliasIn = 1;
}
const bool inplace = aliasIn >= 0;
const int Tr = aliasIn == 0 ? da : aliasIn == 1 ? db : sampleDepth(rng);
SCOPED_TRACE(cv::format("%s caseidx=%d da=%s db=%s Tr=%s a=%sC%d b=%sC%d inplace=%d",
opStr, caseidx, depthToString(da), depthToString(db), depthToString(Tr), shapeStr(sa).c_str(), cn_a,
shapeStr(sb).c_str(), cn_b, inplace ? aliasIn : -1));
Mat a = makeRandom(rng, sa, cn_a, da), b = makeRandom(rng, sb, cn_b, db);
// reference FIRST (an in-place exec may overwrite an input): per output channel pick a's/b's
// channel (C1->Cn broadcast), spatial-broadcast, cast to common type C, op(Tr), then merge.
int C = (da == db) ? da : promote2(da, db);
std::vector<Mat> ach, bch; cv::split(a, ach); cv::split(b, bch);
std::vector<Mat> refch(ocn);
for (int c = 0; c < ocn; c++)
{
Mat apC, bpC;
cvtest::convert(ach[cn_a == 1 ? 0 : c], apC, C);
cvtest::convert(bch[cn_b == 1 ? 0 : c], bpC, C);
Mat aB, bB; cv::broadcast(apC, res, aB); cv::broadcast(bpC, res, bB);
cvtest::add(aB, 1, bB, op == OP_SUB ? -1 : 1, Scalar(), refch[c], Tr);
}
Mat ref; cv::merge(refch, ref);
// public op: dst aliases input #aliasIn for the in-place case, else a fresh Mat.
Mat outOwn;
Mat& out = inplace ? (aliasIn == 0 ? a : b) : outOwn;
if (op == OP_SUB) cv::subtract(a, b, out, noArray(), Tr);
else cv::add (a, b, out, noArray(), Tr);
checkClose(out, ref, Tr, isFloat(da) || isFloat(db), opStr);
}
INSTANTIATE_TEST_CASE_P(Core_EW, EW_Extensive_BinOp,
testing::Combine(testing::Values(0, 1), testing::Range(0, kNumCases)),
[](const testing::TestParamInfo<std::tuple<int,int>>& ti) {
return cv::format("%s_case%04d", std::get<0>(ti.param) ? "sub" : "add",
std::get<1>(ti.param));
});
// ------------------------------------------------------------------- min / max / absdiff
// op 0 = MIN, 1 = MAX, 2 = ABSDIFF: operands promoted to a common type C; MIN/MAX result C, ABSDIFF
// result Cr = absdiffResultDepth(C) (unsigned same width for signed ints, since |a-b| can hit 2^w-1).
// cv::min/max/absdiff auto-promote mixed input types (no dtype arg), so this exercises the engine's
// promotion + cast insertion for a fresh family of ops through the public entry points.
class EW_Extensive_MinMax : public ::testing::TestWithParam<std::tuple<int,int>> {};
TEST_P(EW_Extensive_MinMax, accuracy)
{
const int opSel = std::get<0>(GetParam());
const int caseidx = std::get<1>(GetParam());
const TOp op = opSel == 0 ? OP_MIN : opSel == 1 ? OP_MAX : OP_ABSDIFF;
const char* opStr = opSel == 0 ? "min" : opSel == 1 ? "max" : "absdiff";
RNG rng(mix64(kSuiteSalt ^ (uint64_t)(caseidx * 3 + opSel))); // distinct stream per op
std::vector<int> shape = sampleShape(rng);
const int da = sampleDepth(rng), db = sampleDepth(rng);
const int C = promoteArith(da, db); // common compute/result type (auto), shared with the ref.
const int Cr = C; // min/max/absdiff all resolve to C (absdiff saturates its wide |a-b| back to C)
static const int cncand[] = { 1, 1, 2, 3, 4 };
const int rcn = cncand[rng.uniform(0, 5)];
const int cn_a = rng.uniform(0, 2) ? rcn : 1;
const int cn_b = rng.uniform(0, 2) ? rcn : 1;
const int ocn = std::max(cn_a, cn_b);
std::vector<int> sa(shape.size()), sb(shape.size()), res(shape.size());
for (size_t d = 0; d < shape.size(); d++)
{
sa[d] = rng.uniform(0, 2) ? shape[d] : 1;
sb[d] = rng.uniform(0, 2) ? shape[d] : 1;
res[d] = std::max(sa[d], sb[d]);
}
SCOPED_TRACE(cv::format("%s caseidx=%d da=%d db=%d C=%d a=%sC%d b=%sC%d",
opStr, caseidx, da, db, C, shapeStr(sa).c_str(), cn_a,
shapeStr(sb).c_str(), cn_b));
Mat a = makeRandom(rng, sa, cn_a, da), b = makeRandom(rng, sb, cn_b, db);
// reference: per output channel pick a's/b's channel (C1->Cn), spatial-broadcast, cast to C, op.
std::vector<Mat> ach, bch; cv::split(a, ach); cv::split(b, bch);
std::vector<Mat> refch(ocn);
for (int c = 0; c < ocn; c++)
{
Mat apC, bpC;
cvtest::convert(ach[cn_a == 1 ? 0 : c], apC, C);
cvtest::convert(bch[cn_b == 1 ? 0 : c], bpC, C);
Mat aB, bB; cv::broadcast(apC, res, aB); cv::broadcast(bpC, res, bB);
if (op == OP_MIN) cvtest::min(aB, bB, refch[c]);
else if (op == OP_MAX) cvtest::max(aB, bB, refch[c]);
else cvtest::add(aB, 1, bB, -1, Scalar(), refch[c], Cr, /*calcAbs=*/true);
}
Mat ref; cv::merge(refch, ref);
// public op: min/max/absdiff auto-promote mixed input types to C (= promoteArith(da,db)) and
// broadcast, exactly like the reference; absdiff's result is the unsigned same-width Cr.
Mat out;
if (op == OP_MIN) cv::min(a, b, out);
else if (op == OP_MAX) cv::max(a, b, out);
else cv::absdiff(a, b, out);
checkClose(out, ref, Cr, isFloat(da) || isFloat(db), opStr);
}
INSTANTIATE_TEST_CASE_P(Core_EW, EW_Extensive_MinMax,
testing::Combine(testing::Values(0, 1, 2), testing::Range(0, kNumCases)),
[](const testing::TestParamInfo<std::tuple<int,int>>& ti) {
const int o = std::get<0>(ti.param);
return cv::format("%s_case%04d", o == 0 ? "min" : o == 1 ? "max" : "absdiff",
std::get<1>(ti.param));
});
// ----------------------------------------------------------------------------------- compare
// op 0 = CMP_EQ, 1 = CMP_GT: operands promoted to a common type C, result a u8 mask. Exercises
// emitBinary's compare branch (result forced to u8) and the optional mask value (0/255 default, or
// 0/1 set through TKernel::flags). Inputs are drawn from a small shared range so equality fires.
class EW_Extensive_Compare : public ::testing::TestWithParam<std::tuple<int,int>> {};
TEST_P(EW_Extensive_Compare, accuracy)
{
const int opSel = std::get<0>(GetParam()); // 0 = EQ, 1 = GT
const int caseidx = std::get<1>(GetParam());
const int cmpop = opSel == 0 ? cv::CMP_EQ : cv::CMP_GT;
const char* opStr = opSel == 0 ? "cmpEQ" : "cmpGT";
RNG rng(mix64(kSuiteSalt ^ 0xC0FFEEULL ^ (uint64_t)(caseidx * 2 + opSel)));
std::vector<int> shape = sampleShape(rng);
const int da = sampleDepth(rng), db = sampleDepth(rng);
const int C = (da == db) ? da : promote2(da, db);
static const int cncand[] = { 1, 1, 2, 3, 4 };
const int rcn = cncand[rng.uniform(0, 5)];
const int cn_a = rng.uniform(0, 2) ? rcn : 1;
const int cn_b = rng.uniform(0, 2) ? rcn : 1;
const int ocn = std::max(cn_a, cn_b);
std::vector<int> sa(shape.size()), sb(shape.size()), res(shape.size());
for (size_t d = 0; d < shape.size(); d++)
{
sa[d] = rng.uniform(0, 2) ? shape[d] : 1;
sb[d] = rng.uniform(0, 2) ? shape[d] : 1;
res[d] = std::max(sa[d], sb[d]);
}
SCOPED_TRACE(cv::format("%s caseidx=%d da=%d db=%d C=%d a=%sC%d b=%sC%d",
opStr, caseidx, da, db, C, shapeStr(sa).c_str(), cn_a,
shapeStr(sb).c_str(), cn_b));
// small shared range [0,12] (well within every depth) so EQ is hit on a healthy fraction
Mat a = makeRandom(rng, sa, cn_a, da, 0, 12), b = makeRandom(rng, sb, cn_b, db, 0, 12);
// reference: per channel, cast to C, compare in f64 (exact for these ranges) -> 0/255 mask
std::vector<Mat> ach, bch; cv::split(a, ach); cv::split(b, bch);
std::vector<Mat> refch(ocn);
for (int c = 0; c < ocn; c++)
{
Mat apC, bpC;
cvtest::convert(ach[cn_a == 1 ? 0 : c], apC, C);
cvtest::convert(bch[cn_b == 1 ? 0 : c], bpC, C);
Mat aB, bB; cv::broadcast(apC, res, aB); cv::broadcast(bpC, res, bB);
Mat af, bf; cvtest::convert(aB, af, CV_64F); cvtest::convert(bB, bf, CV_64F);
cvtest::compare(af, bf, refch[c], cmpop); // 0 / 255
}
Mat ref255; cv::merge(refch, ref255);
// public op: cv::compare auto-promotes mixed input types to the common type, broadcasts, and
// yields a u8 0/255 mask per channel. (The engine's optional 0/1 mask is not exposed here.)
Mat out; cv::compare(a, b, out, cmpop);
ASSERT_EQ(out.type(), CV_8UC(ocn)) << opStr;
EXPECT_EQ(0, cvtest::norm(out, ref255, NORM_INF)) << opStr;
}
INSTANTIATE_TEST_CASE_P(Core_EW, EW_Extensive_Compare,
testing::Combine(testing::Values(0, 1), testing::Range(0, kNumCases)),
[](const testing::TestParamInfo<std::tuple<int,int>>& ti) {
return cv::format("%s_case%04d", std::get<0>(ti.param) == 0 ? "cmpEQ" : "cmpGT",
std::get<1>(ti.param));
});
// ------------------------------------------------------------------------------------- mul / div
// Parameterized on (op, caseidx): op 0 = MUL, 1 = DIV. Both compute in the float work type (float
// for <=16-bit, double for 32/64-bit), matching cv::multiply/divide; integer divide-by-zero => 0.
class EW_Extensive_MulDiv : public ::testing::TestWithParam<std::tuple<int,int>> {};
TEST_P(EW_Extensive_MulDiv, accuracy)
{
const int opSel = std::get<0>(GetParam());
const int caseidx = std::get<1>(GetParam());
const TOp op = opSel ? OP_DIV : OP_MUL;
const char* opStr = opSel ? "div" : "mul";
RNG rng(mix64(kSuiteSalt ^ 0x3DD17ULL ^ (uint64_t)caseidx));
std::vector<int> shape = sampleShape(rng);
const int da = sampleDepth(rng), db = sampleDepth(rng);
static const int cncand[] = { 1, 1, 2, 3, 4 };
const int rcn = cncand[rng.uniform(0, 5)];
const int cn_a = rng.uniform(0, 2) ? rcn : 1;
const int cn_b = rng.uniform(0, 2) ? rcn : 1;
const int ocn = std::max(cn_a, cn_b);
std::vector<int> sa(shape.size()), sb(shape.size()), res(shape.size());
for (size_t d = 0; d < shape.size(); d++)
{
sa[d] = rng.uniform(0, 2) ? shape[d] : 1;
sb[d] = rng.uniform(0, 2) ? shape[d] : 1;
res[d] = std::max(sa[d], sb[d]);
}
int aliasIn = -1;
if (rng.uniform(0, 3) == 0)
{
if (sa == res && cn_a == ocn) aliasIn = 0;
else if (sb == res && cn_b == ocn) aliasIn = 1;
}
const bool inplace = aliasIn >= 0;
const int Tr = aliasIn == 0 ? da : aliasIn == 1 ? db : sampleDepth(rng);
// half the cases use a non-unit scale (mul: a*b*scale, div: a*scale/b), like cv::multiply/
// divide. Kept in [1/256, 2] so it can shrink (e.g. 1/255) or modestly amplify without pushing
// a product/quotient past the integer-output range (which would be float->int UB on both sides).
double scale = 1.0;
if (rng.uniform(0, 2)) scale = rng.uniform(1.0/256, 2.0);
SCOPED_TRACE(cv::format("%s caseidx=%d da=%d db=%d Tr=%d a=%sC%d b=%sC%d inplace=%d scale=%.4f",
opStr, caseidx, da, db, Tr, shapeStr(sa).c_str(), cn_a,
shapeStr(sb).c_str(), cn_b, inplace ? aliasIn : -1, scale));
// modest magnitudes: mul/div compute in a float work type, so a product/quotient that overflows
// the integer output's range hits float->int UB (cv::multiply is UB there too). [-1000,1000]
// keeps products <= 1e6 (no overflow), while still exercising saturation for small outputs.
Mat a = makeRandom(rng, sa, cn_a, da, -1000, 1000), b = makeRandom(rng, sb, cn_b, db, -1000, 1000);
const bool bothInt = !isFloat(da) && !isFloat(db);
// Integer divide-by-zero is well-defined (=> 0) and IS exercised. Float-involved divide-by-zero
// is UB (a/0 -> inf -> int), so avoid it here: make the divisor (b) nonzero for the float path.
if (op == OP_DIV && !bothInt)
{
Mat b64; b.convertTo(b64, CV_64F);
b64.setTo(1.0, b64 == 0.0);
b64.convertTo(b, db);
}
// reference FIRST (in-place may overwrite an input): mirror the engine's spec - cast both to the
// float work type Wf (float for <=16-bit common type, double for 32/64-bit), op in Wf, then cast
// to Tr (same final cast the engine uses). For both-integer div, guard divide-by-zero -> 0.
const int C = (da == db) ? da : promote2(da, db);
const bool wide = (C==CV_32U || C==CV_32S || C==CV_64U || C==CV_64S || C==CV_64F);
const int Wf = wide ? CV_64F : CV_32F;
std::vector<Mat> ach, bch; cv::split(a, ach); cv::split(b, bch);
std::vector<Mat> refch(ocn);
for (int c = 0; c < ocn; c++)
{
Mat aWf, bWf;
cvtest::convert(ach[cn_a == 1 ? 0 : c], aWf, Wf);
cvtest::convert(bch[cn_b == 1 ? 0 : c], bWf, Wf);
Mat aB, bB; cv::broadcast(aWf, res, aB); cv::broadcast(bWf, res, bB);
Mat q;
if (op == OP_DIV) { cvtest::divide(aB, bB, q, scale); if (bothInt) q.setTo(0, bB == 0); }
else cvtest::multiply(aB, bB, q, scale);
cvtest::convert(q, refch[c], Tr);
}
Mat ref; cv::merge(refch, ref);
// public op: dst aliases input #aliasIn for the in-place case, else a fresh Mat.
Mat outOwn;
Mat& out = inplace ? (aliasIn == 0 ? a : b) : outOwn;
if (op == OP_DIV) cv::divide (a, b, out, scale, Tr);
else cv::multiply(a, b, out, scale, Tr);
checkClose(out, ref, Tr, true, opStr); // float work => integer output may differ by <=1
}
INSTANTIATE_TEST_CASE_P(Core_EW, EW_Extensive_MulDiv,
testing::Combine(testing::Values(0, 1), testing::Range(0, kNumCases)),
[](const testing::TestParamInfo<std::tuple<int,int>>& ti) {
return cv::format("%s_case%04d", std::get<0>(ti.param) ? "div" : "mul",
std::get<1>(ti.param));
});
// ------------------------------------------------------------------------------- masked add / sub
// add/sub with a write-mask. The data inputs share the output shape (a, b, out all `shape`-spatial,
// cn channels); the mask is single-channel, the output spatial shape, type bool/u8/s8. The output
// PRE-EXISTS (filled with random content): copyMask overwrites only the masked subset and leaves
// the rest unchanged (dst = mask ? op : dst). cn==1 exercises the per-element mask (CH_FOLD); cn>1
// the channel-axis broadcast (CH_DIM, mask stepx 0 => a whole n-channel row copied under one test).
class EW_Extensive_Mask : public ::testing::TestWithParam<std::tuple<int,int>> {};
TEST_P(EW_Extensive_Mask, accuracy)
{
const int opSel = std::get<0>(GetParam());
const int caseidx = std::get<1>(GetParam());
const TOp op = opSel ? OP_SUB : OP_ADD;
const char* opStr = opSel ? "sub" : "add";
RNG rng(mix64(kSuiteSalt ^ 0x5A5C0DEULL ^ (uint64_t)caseidx));
std::vector<int> shape = sampleShape(rng);
const int da = sampleDepth(rng), db = sampleDepth(rng), Tr = sampleDepth(rng);
static const int cncand[] = { 1, 1, 2, 3, 4 };
const int cn = cncand[rng.uniform(0, 5)];
static const int maskDepths[] = { CV_8U, CV_8S, CV_Bool };
const int md = maskDepths[rng.uniform(0, 3)];
SCOPED_TRACE(cv::format("%s caseidx=%d da=%d db=%d Tr=%d cn=%d md=%d shape=%s",
opStr, caseidx, da, db, Tr, cn, md, shapeStr(shape).c_str()));
Mat a = makeRandom(rng, shape, cn, da), b = makeRandom(rng, shape, cn, db);
// mask: single-channel, output spatial shape, ~half zero. Build a u8 0/1 master, convert it to
// the chosen mask depth for the engine; the u8 master drives the reference copyTo.
const int nd = (int)shape.size();
Mat m8(nd, shape.data(), CV_8U);
cvtest::randUni(rng, m8, Scalar::all(0), Scalar::all(2)); // 0 or 1
Mat mask; m8.convertTo(mask, md);
// pre-existing output content (preserved where mask==0): dst = mask ? op : dst.
Mat init = makeRandom(rng, shape, cn, Tr).clone(); // contiguous Tr-typed dst
// reference: full op per channel (cast to common type C, op to Tr), merge, then overwrite the
// masked subset of `init` (the rest stays as the pre-existing content).
int C = (da == db) ? da : promote2(da, db);
std::vector<Mat> ach, bch; cv::split(a, ach); cv::split(b, bch);
std::vector<Mat> refch(cn);
for (int c = 0; c < cn; c++)
{
Mat apC, bpC; cvtest::convert(ach[c], apC, C); cvtest::convert(bch[c], bpC, C);
cvtest::add(apC, 1, bpC, op == OP_SUB ? -1 : 1, Scalar(), refch[c], Tr);
}
Mat refFull; cv::merge(refch, refFull);
Mat ref = init.clone();
cvtest::copy(refFull, ref, m8);
// public op with a write-mask: the pre-existing output is preserved where mask==0.
Mat out = init.clone();
if (op == OP_SUB) cv::subtract(a, b, out, mask, Tr);
else cv::add (a, b, out, mask, Tr);
checkClose(out, ref, Tr, isFloat(da) || isFloat(db), opStr);
}
INSTANTIATE_TEST_CASE_P(Core_EW, EW_Extensive_Mask,
testing::Combine(testing::Values(0, 1), testing::Range(0, kNumCases)),
[](const testing::TestParamInfo<std::tuple<int,int>>& ti) {
return cv::format("%s_case%04d", std::get<0>(ti.param) ? "sub" : "add",
std::get<1>(ti.param));
});
// NOTE: a standalone cast group was dropped - the engine cast == cv::convertTo (comparing them would
// be a tautology), and mixed-type casts are already exercised inside the add/sub/mul/div groups above.
}} // namespace
modules/core/test/test_operations.cpp
+5 -1
View File
@@ -1592,7 +1592,11 @@ TEST(Core_MatExpr, mul_scalar_use_after_scope_23577)
EXPECT_EQ(0, cvtest::norm(res, Mat(2, 3, CV_32FC1, Scalar::all(21.0f)), NORM_INF));
}
TEST(Core_Arithm, scalar_handling_19599) // https://github.com/opencv/opencv/issues/19599 (OpenCV 4.x+ only)
// Disabled with the new broadcasting element-wise engine: a 4x1 CV_64F *Mat* is no longer treated as a
// Scalar (only true scalars - numbers / cv::Scalar / Vec / Matx, which arrive via _InputArray::MATX -
// are scalars; real Mats ride broadcasting). Here b broadcasts against a(1x1) -> 4x1, not 1x1. A
// follow-up OpenCV issue tracks this intended behavior change.
TEST(Core_Arithm, DISABLED_scalar_handling_19599) // https://github.com/opencv/opencv/issues/19599 (OpenCV 4.x+ only)
{
Mat a(1, 1, CV_32F, Scalar::all(1));
Mat b(4, 1, CV_64F, Scalar::all(1)); // MatExpr may convert Scalar to Mat
modules/dnn/include/opencv2/dnn/dict.hpp
+7 -7
View File
@@ -60,13 +60,13 @@ CV__DNN_INLINE_NS_BEGIN
struct CV_EXPORTS_W DictValue
{
DictValue(const DictValue &r);
explicit DictValue(bool i) : type(Param::INT), pi(new AutoBuffer<int64,1>) { (*pi)[0] = i ? 1 : 0; } //!< Constructs integer scalar
explicit DictValue(int64 i = 0) : type(Param::INT), pi(new AutoBuffer<int64,1>) { (*pi)[0] = i; } //!< Constructs integer scalar
CV_WRAP explicit DictValue(int i) : type(Param::INT), pi(new AutoBuffer<int64,1>) { (*pi)[0] = i; } //!< Constructs integer scalar
explicit DictValue(unsigned p) : type(Param::INT), pi(new AutoBuffer<int64,1>) { (*pi)[0] = p; } //!< Constructs integer scalar
CV_WRAP explicit DictValue(double p) : type(Param::REAL), pd(new AutoBuffer<double,1>) { (*pd)[0] = p; } //!< Constructs floating point scalar
CV_WRAP explicit DictValue(const String &s) : type(Param::STRING), ps(new AutoBuffer<String,1>) { (*ps)[0] = s; } //!< Constructs string scalar
explicit DictValue(const char *s) : type(Param::STRING), ps(new AutoBuffer<String,1>) { (*ps)[0] = s; } //!< @overload
explicit DictValue(bool i) : type(Param::INT), pi(new AutoBuffer<int64,1>(1)) { (*pi)[0] = i ? 1 : 0; } //!< Constructs integer scalar
explicit DictValue(int64 i = 0) : type(Param::INT), pi(new AutoBuffer<int64,1>(1)) { (*pi)[0] = i; } //!< Constructs integer scalar
CV_WRAP explicit DictValue(int i) : type(Param::INT), pi(new AutoBuffer<int64,1>(1)) { (*pi)[0] = i; } //!< Constructs integer scalar
explicit DictValue(unsigned p) : type(Param::INT), pi(new AutoBuffer<int64,1>(1)) { (*pi)[0] = p; } //!< Constructs integer scalar
CV_WRAP explicit DictValue(double p) : type(Param::REAL), pd(new AutoBuffer<double,1>(1)) { (*pd)[0] = p; } //!< Constructs floating point scalar
CV_WRAP explicit DictValue(const String &s) : type(Param::STRING), ps(new AutoBuffer<String,1>(1)) { (*ps)[0] = s; } //!< Constructs string scalar
explicit DictValue(const char *s) : type(Param::STRING), ps(new AutoBuffer<String,1>(1)) { (*ps)[0] = s; } //!< @overload
template<typename TypeIter>
static DictValue arrayInt(TypeIter begin, int size); //!< Constructs integer array
modules/dnn/src/layers/batch_norm_layer.cpp
+20 -9
View File
@@ -92,8 +92,12 @@ public:
const float* weightsData = hasWeights ? blobs[weightsBlobIndex].ptr<float>() : 0;
const float* biasData = hasBias ? blobs[biasBlobIndex].ptr<float>() : 0;
origin_weights.create(1, (int)n, CV_32F);
origin_bias.create(1, (int)n, CV_32F);
// 1-D [n], NOT 1xn: the fused scale/bias participate in element-wise ops against 0/1-D
// inputs, and under the broadcasting rules (4) op (1,4) yields (1,4) - a 2-D result that
// would not fit a preallocated 1-D output blob. 1-D weights keep every shape exact.
const int sz1d[] = { (int)n };
origin_weights.create(1, sz1d, CV_32F);
origin_bias.create(1, sz1d, CV_32F);
float* dstWeightsData = origin_weights.ptr<float>();
float* dstBiasData = origin_bias.ptr<float>();
@@ -108,8 +112,8 @@ public:
virtual void finalize(InputArrayOfArrays, OutputArrayOfArrays) CV_OVERRIDE
{
origin_weights.reshape(1, 1).copyTo(weights_);
origin_bias.reshape(1, 1).copyTo(bias_);
origin_weights.copyTo(weights_);
origin_bias.copyTo(bias_);
}
void getScaleShift(Mat& scale, Mat& shift) const CV_OVERRIDE
@@ -133,9 +137,11 @@ public:
(numFusedBias != numChannels && numFusedBias != 1 && !b.empty()))
return false;
// reshape the fused factors to 1-D [numChannels], matching weights_/bias_ - a 1xn operand
// would broadcast the result up to 2-D and detach weights_ from its expected 1-D shape
const int fsz[] = { numChannels };
if (!w.empty())
{
w = w.reshape(1, 1);
if (numFusedWeights == 1)
{
multiply(weights_, w.at<float>(0), weights_);
@@ -143,17 +149,17 @@ public:
}
else
{
w = w.reshape(1, 1, fsz);
multiply(weights_, w, weights_);
multiply(bias_, w, bias_);
}
}
if (!b.empty())
{
b = b.reshape(1, 1);
if (numFusedBias == 1)
add(bias_, b.at<float>(0), bias_);
else
add(bias_, b.reshape(1, 1), bias_);
add(bias_, b.reshape(1, 1, fsz), bias_);
}
return true;
}
@@ -281,8 +287,13 @@ public:
Mat &inpBlob = inputs[0];
Mat &outBlob = outputs[0];
CV_Assert(inpBlob.total() == weights_.total());
cv::multiply(inpBlob, weights_, outBlob);
cv::add(outBlob, bias_, outBlob);
// run on 1-D [n] views (a 0-D blob views as [1]) so every operand shape matches exactly:
// the result lands in the preallocated output blob in place, no realloc/detach possible
const int n1[] = { (int)inpBlob.total() };
Mat inp1d = inpBlob.reshape(1, 1, n1);
Mat out1d = outBlob.reshape(1, 1, n1);
cv::multiply(inp1d, weights_, out1d);
cv::add(out1d, bias_, out1d);
return;
}
modules/dnn/src/layers/recurrent2_layers.cpp
+53 -38
View File
@@ -143,24 +143,34 @@ class LSTM2LayerImpl CV_FINAL : public LSTM2Layer
if (layout == SEQ_BATCH_HID) {
_batchSize = inp0[1];
_seqLen = inp0[0];
outResShape.push_back(_seqLen);
outResShape.push_back(1 + static_cast<int>(bidirectional));
outResShape.push_back(_batchSize);
} else {
// ONNX layout=1: Y is (batch, seq, dirs, hid) - this must match what forward()
// actually writes, the graph engine preallocates the output by this shape
_batchSize = inp0[0];
_seqLen = inp0[1];
outResShape.push_back(_batchSize);
outResShape.push_back(_seqLen);
outResShape.push_back(1 + static_cast<int>(bidirectional));
}
outResShape.push_back(_seqLen);
}
else
{
CV_Assert(inp0.size() >= 2 && total(inp0, 1) == _inpSize);
_batchSize = inp0[0];
outResShape.push_back(1 + static_cast<int>(bidirectional));
outResShape.push_back(_batchSize);
}
outResShape.push_back(1 + static_cast<int>(bidirectional));
outResShape.push_back(_batchSize);
outResShape.push_back(_hidSize);
outputs.assign(1, outResShape);
// Yh / Yc: ONNX layout=0 -> (dirs, batch, hid), layout=1 -> (batch, dirs, hid)
int shp[] = {1 + static_cast<int>(bidirectional), _batchSize, numHidden};
if (layout == BATCH_SEQ_HID)
std::swap(shp[0], shp[1]);
MatShape newShape(shp, shp + sizeof(shp)/sizeof(shp[0]));
// compute output shape of yc
@@ -302,13 +312,16 @@ class LSTM2LayerImpl CV_FINAL : public LSTM2Layer
Mat cOutTs;
Mat cOut = produceCellOutput ? output[0].clone() : Mat();
// seq-major scratch for the cell states, (seq, batch, dirs, hid) like the recurrence writes
int cOutShape[] = {seqLenth, batchSize, numDirs, numHidden};
Mat cOut = produceCellOutput ? Mat::zeros(4, cOutShape, output[0].type()) : Mat();
Mat hOutTs = Mat::zeros(seqLenth * batchSize, hidSize, output[0].type());
Mat xTs = input[0].reshape(1, batchSizeTotal);
// Prepare output[0] buffer to store the results
int shp0[] = {seqLenth * batchSize, numDirs * numHidden};
output[0] = output[0].reshape(1, sizeof(shp0)/sizeof(shp0[0]), shp0);
// seq-major assembly buffer for Y. The final result is transposed from it INTO output[0]:
// the preallocated output tensor must never be reallocated or get its header replaced,
// or the result would silently detach from the graph.
Mat hOutAll(batchSizeTotal, numDirs * numHidden, output[0].type());
// Initialize Wx, Wh, bias, h_0, c_0, pI, pF, pO
Mat Wx, Wh, bias, h_0, c_0, pI, pF, pO;
@@ -416,32 +429,31 @@ class LSTM2LayerImpl CV_FINAL : public LSTM2Layer
}
// slice in the result from each direction to the output[0] buffer
hOutTs.copyTo(output[0].colRange(i * hOutTs.cols, (i + 1) * hOutTs.cols));
// slice in the result from each direction to the assembly buffer
hOutTs.copyTo(hOutAll.colRange(i * hOutTs.cols, (i + 1) * hOutTs.cols));
}
// this one is needed to make make the output[0] compatible with ONNX LSTM layer standard
// (seq*batch, dirs*hid) -> (seq, batch, dirs, hid), then into the ONNX Y layout for this
// `layout` attribute, written INTO the preallocated output[0] (transposeND's exact-shape
// create() keeps it in place)
int shp1[] = {seqLenth, batchSize, numDirs, numHidden};
output[0] = output[0].reshape(1, sizeof(shp1)/sizeof(shp1[0]), shp1);
// this transpose is needed to make the output[0] compatible with ONNX LSTM layer standard
Mat tmp = output[0].clone();
cv::transposeND(tmp, {0, 2, 1, 3}, output[0]);
Mat y4d = hOutAll.reshape(1, sizeof(shp1)/sizeof(shp1[0]), shp1);
Mat ySeqFirst; // (seq, dirs, batch, hid) - the layout=0 Y; Yh is sliced from it
if (layout == SEQ_BATCH_HID) {
cv::transposeND(y4d, {0, 2, 1, 3}, output[0]);
ySeqFirst = output[0];
} else {
cv::transposeND(y4d, {0, 2, 1, 3}, ySeqFirst);
cv::transposeND(y4d, {1, 0, 2, 3}, output[0]); // (batch, seq, dirs, hid)
}
if (produceOutputYh){
getCellStateYh(output[0], output[1], numDirs);
getCellStateYh(ySeqFirst, output[1], numDirs);
}
if (produceCellOutput){
getCellStateYc(cOut, output[2], numDirs);
}
if (layout == BATCH_SEQ_HID) {
cv::transposeND(output[0], {2, 0, 1, 3}, output[0]);
}
// Make sure changes are written back to outputs_arr
outputs_arr.assign(output);
}
void getCellStateYh(Mat& scr, Mat& dst, int numDirs)
@@ -463,28 +475,30 @@ class LSTM2LayerImpl CV_FINAL : public LSTM2Layer
}
} else {
// there is issue here.
// Slice: SxDxBxH -> last sequence, first direction
Range ranges1[] = {cv::Range(scr.size[0] - 1, scr.size[0]), cv::Range(0, 1), cv::Range::all(), cv::Range::all()};
Mat part1 = scr(ranges1);
// Slice: SxDxBxH -> first sequence, last direction
Range ranges2[] = {cv::Range(0, 1), cv::Range(scr.size[1] - 1, scr.size[1]), cv::Range::all(), cv::Range::all()};
Mat part2 = scr(ranges2);
int shp[] = {1, part1.size[2] * part1.size[3]};
part1 = part1.reshape(1, sizeof(shp)/sizeof(shp[0]), shp);
part2 = part2.reshape(1, sizeof(shp)/sizeof(shp[0]), shp);
vconcat(part1, part2, dst);
// build into a temp, then write into the preallocated dst in place (vconcat straight
// into dst would replace its header and detach it from the graph's output tensor)
Mat tmp;
vconcat(part1, part2, tmp);
int finalShape[] = {2, batchSize, numHidden};
dst = dst.reshape(1, sizeof(finalShape)/sizeof(finalShape[0]), finalShape);
tmp = tmp.reshape(1, sizeof(finalShape)/sizeof(finalShape[0]), finalShape);
if (layout == BATCH_SEQ_HID){
cv::transposeND(dst, {1, 0, 2}, dst);
cv::transposeND(tmp, {1, 0, 2}, dst);
} else {
tmp.copyTo(dst);
}
}
}
@@ -496,15 +510,10 @@ class LSTM2LayerImpl CV_FINAL : public LSTM2Layer
int shp[] = {0, batchSize, numDirs, numHidden};
cOut = cOut.reshape(1, sizeof(shp)/sizeof(shp[0]), shp);
// permute to {0, 2, 1, 3};
// permute to (seq, dirs, batch, hidden); the `layout` only affects the FINAL Yc order
// below, the last-timestep/last-direction slicing is layout-independent
cv::Mat newCellState;
// transpose to match batch first output
if (layout == BATCH_SEQ_HID){
cv::transposeND(cOut, {2, 0, 1, 3}, newCellState);
}
else{
cv::transposeND(cOut, {0, 2, 1, 3}, newCellState);
}
cv::transposeND(cOut, {0, 2, 1, 3}, newCellState);
cOut = newCellState;
if (numDirs == 1)
@@ -515,7 +524,6 @@ class LSTM2LayerImpl CV_FINAL : public LSTM2Layer
// Reshape: 1x1xBxH -> 1xBxH
int shp[] = {1, batchSize, numHidden};
cOut = cOut.reshape(1, sizeof(shp)/sizeof(shp[0]), shp);
cOut.copyTo(dst);
}
else
{
@@ -536,6 +544,13 @@ class LSTM2LayerImpl CV_FINAL : public LSTM2Layer
// Reshape: 1x2xBxH -> 2xBxH
int finalShape[] = {2, batchSize, numHidden};
cOut = cOut.reshape(1, sizeof(finalShape)/sizeof(finalShape[0]), finalShape);
}
// (dirs, batch, hid), or (batch, dirs, hid) for the batch-first layout - written into the
// preallocated dst in place
if (layout == BATCH_SEQ_HID){
cv::transposeND(cOut, {1, 0, 2}, dst);
} else {
cOut.copyTo(dst);
}
}
modules/dnn/src/layers/recurrent_layers.cpp
+22 -7
View File
@@ -314,8 +314,12 @@ public:
if (layout == SEQ_BATCH_HID) {
_numSamples = inp0[1];
outResShape.push_back(inp0[0]);
outResShape.push_back(_numSamples);
} else {
// batch-first layout: the output keeps the (batch, seq, ...) order - this must match
// what forward() actually writes, the graph engine preallocates outputs by this shape
_numSamples = inp0[0];
outResShape.push_back(_numSamples);
outResShape.push_back(inp0[1]);
}
}
@@ -323,9 +327,9 @@ public:
{
CV_Assert(inp0.size() >= 2 && total(inp0, 1) == _numInp);
_numSamples = inp0[0];
outResShape.push_back(_numSamples);
}
outResShape.push_back(_numSamples);
outResShape.insert(outResShape.end(), outTailShape_.begin(), outTailShape_.end());
outResShape.back() *= (1 + static_cast<int>(bidirectional));
@@ -428,7 +432,19 @@ public:
input[0] = tmp;
}
Mat cOut = produceCellOutput ? output[0].clone() : Mat();
// For the batch-first layout the (preallocated) output[0] is (batch, seq, ...), but the
// recurrence below assembles rows in seq-major order - run it on a seq-first temp, then
// transpose INTO output[0] at the end. output[0] itself must never be reallocated/replaced:
// it is the tensor the graph engine preallocated, a new header would silently detach from it.
Mat hOutSeqFirst = output[0];
if (layout == BATCH_SEQ_HID)
{
MatShape seqFirstShape = output[0].shape();
std::swap(seqFirstShape[0], seqFirstShape[1]);
hOutSeqFirst = Mat(seqFirstShape, output[0].type());
}
Mat cOut = produceCellOutput ? hOutSeqFirst.clone() : Mat();
const bool needYcTransform = !originalBlobs.empty(); // if the producer is onnx
const int numDirs = 1 + static_cast<int>(bidirectional);
for (int i = 0; i < numDirs; ++i)
@@ -484,7 +500,7 @@ public:
int numSamplesTotal = numTimeStamps*numSamples;
Mat xTs = input[0].reshape(1, numSamplesTotal);
Mat hOutTs = output[0].reshape(1, numSamplesTotal);
Mat hOutTs = hOutSeqFirst.reshape(1, numSamplesTotal);
hOutTs = hOutTs.colRange(i * hOutTs.cols / numDirs, (i + 1) * hOutTs.cols / numDirs);
Mat cOutTs;
if (produceCellOutput)
@@ -727,11 +743,10 @@ public:
cInternal.copyTo(cOutTs.rowRange(curRowRange));
}
}
// transpose to match batch first output
// transpose to match batch first output - into the preallocated tensor (exact-shape create()
// inside transposeND reuses it, so the graph's output blob is written in place)
if (layout == BATCH_SEQ_HID){
cv::Mat tmp;
cv::transposeND(output[0], {1, 0, 2}, tmp);
output[0] = tmp;
cv::transposeND(hOutSeqFirst, {1, 0, 2}, output[0]);
}
if (needYcTransform && produceCellOutput)
{
modules/dnn/src/net_impl2.cpp
+18
View File
@@ -1342,6 +1342,24 @@ void Net::Impl::forwardGraph(Ptr<Graph>& graph, InputArrayOfArrays inputs_,
m.copyTo(buf);
}
} else {
if (!dynamicOutShapes) {
// the same sanity check for non-temp (graph output/state) tensors: the layer must
// write into the preallocated tensor of the inferred shape/type. A mismatch here
// means some op inside Layer::forward() reallocated it (e.g. a broadcast produced
// an unexpected shape) and the result would silently detach from the graph.
if (m.shape() != outShapes[i] || m.type() != outTypes[i] ||
(m.u && (m.u->data != outOrigData[i].first || m.u->size != outOrigData[i].second)))
{
std::ostringstream oss;
oss << "layer '" << layer->name << "' (" << layer->type << "): output #" << i
<< " changed during forward(): inferred shape " << outShapes[i]
<< " / type " << typeToString(outTypes[i])
<< ", actual " << m.shape() << " / " << typeToString(m.type())
<< (m.u && m.u->data != outOrigData[i].first
? "; the tensor was reallocated (the layer must write in place)" : "");
CV_Error(Error::StsInternal, oss.str());
}
}
__tensors__.at(out.idx) = m;
}
}
modules/features/src/blobdetector.cpp
+14 -8
View File
@@ -47,7 +47,7 @@
#include <opencv2/core/utils/logger.hpp>
// Requires CMake flag: DEBUG_opencv_features=ON
//#define DEBUG_BLOB_DETECTOR
// #define DEBUG_BLOB_DETECTOR
#ifdef DEBUG_BLOB_DETECTOR
#include "opencv2/highgui.hpp"
@@ -253,6 +253,8 @@ void SimpleBlobDetectorImpl::findBlobs(InputArray _image, InputArray _binaryImag
imshow("contours", contoursImage );
#endif
std::vector<double> dists;
for (size_t contourIdx = 0; contourIdx < contours.size(); contourIdx++)
{
Center center;
@@ -317,6 +319,7 @@ void SimpleBlobDetectorImpl::findBlobs(InputArray _image, InputArray _binaryImag
if(moms.m00 == 0.0)
continue;
center.location = Point2d(moms.m10 / moms.m00, moms.m01 / moms.m00);
center.radius = 0.;
if (params.filterByColor)
{
@@ -328,14 +331,17 @@ void SimpleBlobDetectorImpl::findBlobs(InputArray _image, InputArray _binaryImag
{
const std::vector<cv::Point>& contour = contours[contourIdx];
const size_t contourSize = contour.size();
AutoBuffer<double> dists(contourSize);
for (size_t pointIdx = 0; pointIdx < contourSize; pointIdx++)
{
const Point2d& pt = contour[pointIdx];
dists[pointIdx] = norm(center.location - pt);
if (contourSize > 0u) {
dists.resize(contourSize);
for (size_t pointIdx = 0; pointIdx < contourSize; pointIdx++)
{
const Point2d& pt = contour[pointIdx];
dists[pointIdx] = norm(center.location - pt);
}
std::sort(dists.begin(), dists.end());
center.radius = (dists[(contourSize - 1) / 2] + dists[contourSize / 2]) / 2.;
}
std::sort(dists.begin(), dists.end());
center.radius = (dists[(dists.size() - 1) / 2] + dists[dists.size() / 2]) / 2.;
}
centers.push_back(center);
modules/python/test/test_expr.py
+103
View File
@@ -0,0 +1,103 @@
#!/usr/bin/env python
'''
cv.texpr() - the string front-end of the broadcasting element-wise expression engine
'''
# Python 2/3 compatibility
from __future__ import print_function
import numpy as np
import cv2 as cv
from tests_common import NewOpenCVTests
class texpr_test(NewOpenCVTests):
def test_basic_arith(self):
a = np.random.uniform(1, 10, (12, 15)).astype(np.float32)
b = np.random.uniform(1, 10, (12, 15)).astype(np.float32)
res = cv.texpr("{0} * 2.5 + {1}", [a, b])
self.assertIsInstance(res, tuple)
self.assertEqual(len(res), 1)
self.assertLessEqual(np.max(np.abs(res[0] - (a * 2.5 + b))), 1e-3)
# the idiomatic single-result unpacking
r, = cv.texpr("{0} * 2.5 + {1}", [a, b])
self.assertTrue(np.array_equal(r, res[0]))
def test_fused_absdiff(self):
a = np.random.randint(0, 255, (20, 30)).astype(np.uint8)
b = np.random.randint(0, 255, (20, 30)).astype(np.uint8)
got = cv.texpr("abs({0} - {1})", [a, b])[0]
self.assertEqual(got.dtype, np.uint8)
self.assertTrue(np.array_equal(got, cv.absdiff(a, b)))
def test_type_cast(self):
a = np.random.uniform(-50, 300, (10, 10)).astype(np.float32)
got = cv.texpr("uint8({0})", [a])[0]
self.assertEqual(got.dtype, np.uint8)
ref = np.clip(np.rint(a), 0, 255).astype(np.uint8)
self.assertTrue(np.array_equal(got, ref))
def test_broadcasting(self):
img = np.random.uniform(0, 255, (8, 6, 3)).astype(np.float32)
row = np.random.uniform(1, 2, (1, 6, 3)).astype(np.float32)
got = cv.texpr("{0} * {1}", [img, row])[0]
self.assertLessEqual(np.max(np.abs(got - img * row)), 1e-3)
def test_ternary_and_compare(self):
a = np.random.uniform(0, 100, (9, 14)).astype(np.float32)
b = np.random.uniform(0, 100, (9, 14)).astype(np.float32)
got = cv.texpr("{0} > {1} ? {0} : {1}", [a, b])[0]
self.assertTrue(np.array_equal(got, np.maximum(a, b)))
def test_pow_operator(self):
a = np.random.uniform(0.5, 2, (7, 11)).astype(np.float32)
got = cv.texpr("3 * {0} ** 2", [a])[0] # ** binds tighter than *
self.assertLessEqual(np.max(np.abs(got - 3 * a ** 2) / (3 * a ** 2)), 1e-5)
def test_math_functions(self):
x = np.random.uniform(0.05, 9, (13, 17)).astype(np.float32)
got = cv.texpr("exp(-{0}) + log({0}) + sqrt({0})", [x])[0]
ref = np.exp(-x.astype(np.float64)) + np.log(x.astype(np.float64)) + np.sqrt(x.astype(np.float64))
self.assertLessEqual(np.max(np.abs(got - ref)), 1e-4)
def test_clamp_scalar_literals(self):
a = np.random.uniform(-100, 355, (10, 21)).astype(np.float32)
got = cv.texpr("clamp({0}, 10, 200)", [a])[0]
self.assertTrue(np.array_equal(got, np.clip(a, 10, 200)))
def test_named_temporary(self):
a = np.random.uniform(1, 10, (6, 8)).astype(np.float32)
b = np.random.uniform(1, 10, (6, 8)).astype(np.float32)
got = cv.texpr("d = {0} - {1}; d*d", [a, b])[0]
self.assertLessEqual(np.max(np.abs(got - (a - b) ** 2)), 1e-3)
def test_tuple_outputs(self):
a = np.random.uniform(1, 10, (5, 9)).astype(np.float32)
b = np.random.uniform(1, 10, (5, 9)).astype(np.float32)
res = cv.texpr("({0} + {1}, {0} - {1})", [a, b])
self.assertEqual(len(res), 2)
self.assertLessEqual(np.max(np.abs(res[0] - (a + b))), 1e-3)
self.assertLessEqual(np.max(np.abs(res[1] - (a - b))), 1e-3)
def test_cart_to_polar(self):
x = np.random.uniform(-5, 5, (11, 13)).astype(np.float32)
y = np.random.uniform(-5, 5, (11, 13)).astype(np.float32)
mag, ang = cv.texpr("(hypot({0},{1}), atan2({1},{0}))", [x, y])
self.assertLessEqual(np.max(np.abs(mag - np.hypot(x, y))), 1e-3)
self.assertLessEqual(np.max(np.abs(ang - np.arctan2(y, x))), 2e-4)
def test_int_saturation(self):
# 32-bit add/subtract saturate (the new v_add_sat kernels)
a = np.array([[2**31 - 1, -2**31, 0]], dtype=np.int32)
b = np.array([[1, -1, -2**31]], dtype=np.int32)
s = cv.texpr("{0} + {1}", [a, b])[0]
d = cv.texpr("{0} - {1}", [a, b])[0]
self.assertEqual(s[0, 0], 2**31 - 1) # MAX + 1 -> MAX
self.assertEqual(s[0, 1], -2**31) # MIN - 1 -> MIN
self.assertEqual(d[0, 2], 2**31 - 1) # 0 - MIN -> MAX
if __name__ == '__main__':
NewOpenCVTests.bootstrap()