mirror of
https://github.com/opencv/opencv.git
synced 2026-09-26 04:09:43 +03:00
5.x
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jaivardhan Bhola
|
84c2360c27 |
Merge pull request #29675 from jaivardhan-bhola:generalized-tokenizer
Generalize tokenizer loading to support method-based family dispatch in DNN - #29675 Companion PR: https://github.com/opencv/opencv_extra/pull/1402 ### Changes Added ALBERT and BERT support end-to-end, with `samples/dnn/albert_inference.py `and `samples/dnn/bert_inference.py `as validation samples, plus expanded coverage in modules/dnn/test/test_tokenizer.cpp. Required changes to `cv::dnn::dnn.hpp`, `graph_fusion_attention.cpp`, and `unicode.cpp/unicode.hpp` to support Unigram and WordPiece tokenizers. To back ALBERT/BERT, generalized `cv::dnn::Tokenizer `from a single BPE implementation into a method-dispatched frontend, adding `core_wordpiece.cpp/hpp` (WordPiece) and `core_unigram.cpp/hpp` (Unigram) as new backends. `tokenizer.cpp` now routes by method across BPE, Gemma,, SentencePiece, Unigram, and WordPiece behind one shared interface. Tested against the following samples and the output matches to old tokenizer: ``` gpt2_inference.py qwen_inference.py gemma3_inference.py ``` GPT2: ``` Preparing GPT-2 model... Inferencing GPT-2 model... Hello, I'm a language model, not a programming language. I'm a language model. I'm a language model. I'm a language model. I'm a language model. I'm a ``` Gemma3: ``` Preparing Gemma3 model... Prompt: <start_of_turn>user What is OpenCV?<end_of_turn> <start_of_turn>model Inferencing Gemma3 model... Response: Okay, let's break down what OpenCV is. **What is OpenCV?** OpenCV (Open Source Computer Vision Library) is a powerful and ``` Qwen2.5: ``` Preparing Qwen2.5 model... Prompt: <|im_start|>user What is OpenCV?<|im_end|> <|im_start|>assistant Inferencing Qwen2.5 model... Response: OpenCV is a set of computer vision libraries in C++ designed to be used for image and video processing. It provides a wide range of tools and functions for ``` ### Tokenizer References Byte-level BPE: [tokenizers/src/pre_tokenizers/byte_level.rs](https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/byte_level.rs) (This defines the byte-level mapping rules, which is used in conjunction with the [BPE model](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/bpe/mod.rs)) SentencePiece BPE (Metaspace): [tokenizers/src/pre_tokenizers/metaspace.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/metaspace.rs) (This defines the rule for replacing whitespace with the U+2581 _ character and handling byte fallback) Unigram: [tokenizers/src/models/unigram/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/unigram/mod.rs) (This contains the core logic for the Unigram lattice scoring and probabilistic tokenization rules) WordPiece: [tokenizers/src/models/wordpiece/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/wordpiece/mod.rs) (This explicitly cites Schuster & Nakajima in the code comments and implements the greedy longest-match rule with the ## prefix) ### Pull Request Readiness Checklist - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on code under GPL or another license incompatible with OpenCV. - [x] The PR is proposed to the proper branch (`5.x`). - [x] There is a reference to the original bug report and related work. - [x] There is accuracy test and test data in `opencv_extra`, same branch name (`generalized-tokenizer`) — `bert/`, `t5/` fixtures back the new C++ tests. - [x] The feature is documented and sample code builds with project CMake. |
||
 Prasad Ayush Kumar
|
fb8afc53c9 |
Merge pull request #29702 from Prasadayus:test_suite_cleanup
Re-enable & triage DISABLED tests for DNN module - #29702 Requires: https://github.com/opencv/opencv_extra/pull/1403 **co-authored by: @varun-jaiswal17** ### PR Changes: ## dnn test cleanup: re-enable stale-disabled tests, fix defect-blind tests, remove redundant coverage ### Removed (dead or unbuildable) - `test_int8_layers.cpp` (1118 lines, removed entirely): cannot compile — `Net::quantize()`, `getInputDetails`/`getOutputDetails` are all gone from `dnn.hpp`/`dnn/src`. Disabled in that same PR (#24980) because on-the-fly quantization was removed — every test in this file called `net.quantize()` to calibrate and run its own int8 conversion. Its own header comment said restore "when test models are quantized outside OpenCV". Pre-quantized ONNX/TFLite test models already do that. ### Removed (redundant or assertion-free) - `Tokenizer_BPE.Tokenizer_GPT2_Model`: line-for-line subset of `Tokenizer_GPT2` — same config, same input, same roundtrip assertion. - `Test_TensorFlow.read_inception`: printed `out.dims` and asserted nothing about the result; `inception_accuracy` loads the same `.pb` and checks it against a reference. - `Test_Caffe_nets` fixture + `INSTANTIATE`: registered **zero** `TEST_P` cases — dead scaffolding for Faster R-CNN tests removed earlier. - `Test_ONNX_nets.Squeezenet`: kernels {1×1, 3×3} and every op type already covered by dedicated layer tests. - `Test_ONNX_nets.VGG16_bn`: single conv kernel (3×3), fully covered by dedicated layer tests; skipped by default anyway under `mem_6gb`. - `Test_ONNX_nets.CaffeNet`: identical op multiset, node count (24) and conv signatures to retained `Alexnet`. - `Test_ONNX_nets.RCNN_ILSVRC13`: `Alexnet` minus `Softmax` (23 vs 24 nodes), identical conv signatures. - `Test_ONNX_nets.Inception_v1`: same op set as retained `Googlenet` (+1 `Reshape`) — Inception v1 *is* GoogLeNet. ### Given real assertions instead of stale expectations - `Test_ONNX_layers.Elementwise_Sqrt`: moved `testONNXModels("sqrt")` below `#endif` — its only work line sat inside `INF_ENGINE_VER_MAJOR_LT(2021040000)`, so without OpenVINO the body compiled to nothing and reported `[ OK ]` on all 3 backends. - `Layer_Test_01D.Clip`: now calls `ClipLayer::create` with `"min"`/`"max"` — it set `lp.type = "Clip"` but constructed `ReLU6Layer::create`, and `runLayer` never reads `layer->type`, so it just re-ran `ReLU6`. - `Layer_Arg_Test`: removed the "disabled" comment, corrected the `convertTo` comment — the comment said the test was disabled while it runs 8 cases, and the second said "convert to float" where the code converts to `CV_64S`. ### Re-enabled as-is (stale disable reasons) - `Test_ONNX_layers.LSTM`/`LSTM_bidirectional` (`test_onnx_importer.cpp:1551,1558`): disabled by #21522 (2022) for poor 1-D-mat handling in the importer of that era; no longer reproduces. - `Test_ONNX_layers.Split_sizes_0d` (`:1373`): disabled by #22652 for a Mul/0-d-tensor shape ambiguity (A×1 vs 1×A); dnn now supports real 1-D Mats, so the output matches the reference exactly. - `DNNTestNetwork.YOLOv8n` ### Library fixes found while re-enabling - `Test_ONNX_layers.LSTM_layout_seq`/`LSTM_layout_batch` (`test_onnx_importer.cpp:1721,1728`): `LSTM2` never transposed `X` for ONNX `layout=1` (batch-first); fixed via `transposeND` gated on `layout==BATCH_SEQ_HID` (`recurrent2_layers.cpp:172`). Fixture also had a leaked loop variable that made the reference a copy of the input; rebuilt by hand since ORT itself refuses to run `layout=1`. - `Test_Graph_Simplifier.ResizeSubgraph` (`test_graph_simplifier.cpp:61`): disabled by the block-layout PR #28585; expectations updated for the `TransformLayout` pass that PR introduced. The test now covers 4 subgraphs rather than 6, because `GatherCastSubgraph` and `MulCastSubgraph` were removed by `0e36cafcf4` and `7669897910` (`Gather`/`Mul` -> `Cast` is no longer fused, since folding it away silently dropped the `Cast`'s dtype semantics). The dynamic-scale `Shape`/`Gather`/`Cast`/`Floor`/`Concat`/`Unsqueeze`/`Slice` chain these models use to compute Resize's scale factor therefore no longer collapses, and the `Mul` survives as `NaryEltwise`, which is why the expected layer lists grew ### Deliberately kept - `ZFNet`: its **7×7** conv appears in no dedicated layer test, and its kernel set {7×7, 5×5, 3×3} differs from `Alexnet`'s {11×11, 5×5, 3×3}. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake |
||
 vrooomy
|
d2c2a82cc9 | add gemma2 SentencePiece tokenizer support | ||
 Varun Jaiswal
|
faf34f95af |
Merge pull request #28837 from varun-jaiswal17:gemma3-tokenizer
Add Gemma3 tokenizer support for dnn #28837 - Adds Gemma3 tokenizer support - Implements character-level BPE - Adds 6 tests covering English, phrase, mixed case, numbers, special tokens, and encode/decode - add gemma3_inference.py Merge with: - **Companion PR** : https://github.com/opencv/opencv_extra/pull/1346 - forward pass bug in gemma3_inference.py : https://github.com/opencv/opencv/pull/28836 ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake |
||
|
Varun Jaiswal
|
c0ab6b69a8 |
Merge pull request #28781 from varun-jaiswal17:work-next
Add Qwen2.5 tokenizer support for dnn #28781 OpenCV extra: https://github.com/opencv/opencv_extra/pull/1334 Extended the dnn tokenizer support to qwen2.5 tokenization. - Add QWEN2_5 pre-tokenizer regex pattern to utils.hpp - Generalised buildTokenizerGPT to buildTokenizerFromJson to handle gpt2/gpt4/ and qwen2.5 - Add qwen2/qwen2.5 model type support with special token handling - Add Qwen2.5 tests - Add end-to-end qwen_inference script for Qwen2.5 ONNX model ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake |
||
 Jorge Velez
|
a49a293d3c |
Merge pull request #27534 from JorgeV92:gsoc2025-tokenizer
GSoC 2025: Add Tokenizer Support to DNN Module #27534 merge with https://github.com/opencv/opencv_extra/pull/1276 ### Summary This pull request introduces initial support for a tokenizer module under `modules/dnn/src/tokenizer` as part of Google Summer of Code 2025 (Project: Tokenization for OpenCV DNN). ### Status - [x] Project structure in place - [x] Initial BPE tokenizer loading - [x] Regex splitting (in progress) - [x] Encoding logic for GPT-2 tokenizer (in progress) - [ ] Documentation (to be improved) ### Goals The goal is to support Hugging Face-compatible tokenization (e.g., GPT-2) natively in C++ to be integrated with DNN inference pipelines. The core pipeline lives in `dnn/src/tokenizer/core_bpe.hpp` and `dnn/src/tokenizer/encoding.hpp`. For Unicode handling I’m using `dnn/src/tokenizer/unicode.hpp`, which is adapted from llama.cpp. ### Feedback Please share early feedback on: - General design structure - Integration strategy with `dnn` - Code organization or naming conventions ### Reference Project: https://summerofcode.withgoogle.com/programs/2025/projects/79SW6eNK |