mirror of
https://github.com/opencv/opencv.git
synced 2026-09-25 04:09:57 +03:00
Jaivardhan Bhola
84c2360c27
Generalize tokenizer loading to support method-based family dispatch in DNN - #29675 Companion PR: https://github.com/opencv/opencv_extra/pull/1402 ### Changes Added ALBERT and BERT support end-to-end, with `samples/dnn/albert_inference.py `and `samples/dnn/bert_inference.py `as validation samples, plus expanded coverage in modules/dnn/test/test_tokenizer.cpp. Required changes to `cv::dnn::dnn.hpp`, `graph_fusion_attention.cpp`, and `unicode.cpp/unicode.hpp` to support Unigram and WordPiece tokenizers. To back ALBERT/BERT, generalized `cv::dnn::Tokenizer `from a single BPE implementation into a method-dispatched frontend, adding `core_wordpiece.cpp/hpp` (WordPiece) and `core_unigram.cpp/hpp` (Unigram) as new backends. `tokenizer.cpp` now routes by method across BPE, Gemma,, SentencePiece, Unigram, and WordPiece behind one shared interface. Tested against the following samples and the output matches to old tokenizer: ``` gpt2_inference.py qwen_inference.py gemma3_inference.py ``` GPT2: ``` Preparing GPT-2 model... Inferencing GPT-2 model... Hello, I'm a language model, not a programming language. I'm a language model. I'm a language model. I'm a language model. I'm a language model. I'm a ``` Gemma3: ``` Preparing Gemma3 model... Prompt: <start_of_turn>user What is OpenCV?<end_of_turn> <start_of_turn>model Inferencing Gemma3 model... Response: Okay, let's break down what OpenCV is. **What is OpenCV?** OpenCV (Open Source Computer Vision Library) is a powerful and ``` Qwen2.5: ``` Preparing Qwen2.5 model... Prompt: <|im_start|>user What is OpenCV?<|im_end|> <|im_start|>assistant Inferencing Qwen2.5 model... Response: OpenCV is a set of computer vision libraries in C++ designed to be used for image and video processing. It provides a wide range of tools and functions for ``` ### Tokenizer References Byte-level BPE: [tokenizers/src/pre_tokenizers/byte_level.rs](https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/byte_level.rs) (This defines the byte-level mapping rules, which is used in conjunction with the [BPE model](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/bpe/mod.rs)) SentencePiece BPE (Metaspace): [tokenizers/src/pre_tokenizers/metaspace.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/metaspace.rs) (This defines the rule for replacing whitespace with the U+2581 _ character and handling byte fallback) Unigram: [tokenizers/src/models/unigram/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/unigram/mod.rs) (This contains the core logic for the Unigram lattice scoring and probabilistic tokenization rules) WordPiece: [tokenizers/src/models/wordpiece/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/wordpiece/mod.rs) (This explicitly cites Schuster & Nakajima in the code comments and implements the greedy longest-match rule with the ## prefix) ### Pull Request Readiness Checklist - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on code under GPL or another license incompatible with OpenCV. - [x] The PR is proposed to the proper branch (`5.x`). - [x] There is a reference to the original bug report and related work. - [x] There is accuracy test and test data in `opencv_extra`, same branch name (`generalized-tokenizer`) — `bert/`, `t5/` fixtures back the new C++ tests. - [x] The feature is documented and sample code builds with project CMake.
OpenCV deep learning module samples
Model Zoo
Check a wiki for a list of tested models.
If OpenCV is built with Intel's Inference Engine support you can use Intel's pre-trained models.
There are different preprocessing parameters such mean subtraction or scale factors for different models. You may check the most popular models and their parameters at models.yml configuration file. It might be also used for aliasing samples parameters. In example,
python object_detection.py opencv_fd --model /path/to/model.onnx
Check -h option to know which values are used by default:
python object_detection.py opencv_fd -h
Sample models
You can download sample models using download_models.py. For example, the following command will download network weights for OpenCV Face Detector model and store them in FaceDetector folder:
python download_models.py --save_dir FaceDetector opencv_fd
You can use default configuration files adopted for OpenCV from here.
You also can use the script to download necessary files from your code. Assume you have the following code inside your_script.py:
from download_models import downloadFile
filepath1 = downloadFile("https://huggingface.co/onnxmodelzoo/ssd_mobilenet_v1_12/resolve/main/ssd_mobilenet_v1_12.onnx", None, filename="ssd_mobilenet_v1_12.onnx", save_dir="save_dir_1")
filepath2 = downloadFile("https://huggingface.co/onnxmodelzoo/ssd_mobilenet_v1_12/resolve/main/ssd_mobilenet_v1_12.onnx", "83536889adce1eda154175f8e3b156dd20443631", filename="ssd_mobilenet_v1_12.onnx")
print(filepath1)
print(filepath2)
# Your code
By running the following commands, you will get ssd_mobilenet_v1_12.onnx file:
export OPENCV_DOWNLOAD_DATA_PATH=download_folder
python your_script.py
Note that you can provide a directory using save_dir parameter or via OPENCV_SAVE_DIR environment variable.
Face detection
An origin model
with single precision floating point weights has been quantized using TensorFlow framework.
To achieve the best accuracy run the model on BGR images resized to 300x300 applying mean subtraction
of values (104, 177, 123) for each blue, green and red channels correspondingly.
The following are accuracy metrics obtained using COCO object detection evaluation
tool on FDDB dataset
(see script)
applying resize to 300x300 and keeping an origin images' sizes.
AP - Average Precision | FP32/FP16 | UINT8 | FP32/FP16 | UINT8 |
AR - Average Recall | 300x300 | 300x300 | any size | any size |
--------------------------------------------------|-----------|----------------|-----------|----------------|
AP @[ IoU=0.50:0.95 | area= all | maxDets=100 ] | 0.408 | 0.408 | 0.378 | 0.328 (-0.050) |
AP @[ IoU=0.50 | area= all | maxDets=100 ] | 0.849 | 0.849 | 0.797 | 0.790 (-0.007) |
AP @[ IoU=0.75 | area= all | maxDets=100 ] | 0.251 | 0.251 | 0.208 | 0.140 (-0.068) |
AP @[ IoU=0.50:0.95 | area= small | maxDets=100 ] | 0.050 | 0.051 (+0.001) | 0.107 | 0.070 (-0.037) |
AP @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] | 0.381 | 0.379 (-0.002) | 0.380 | 0.368 (-0.012) |
AP @[ IoU=0.50:0.95 | area= large | maxDets=100 ] | 0.455 | 0.455 | 0.412 | 0.337 (-0.075) |
AR @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] | 0.299 | 0.299 | 0.279 | 0.246 (-0.033) |
AR @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] | 0.482 | 0.482 | 0.476 | 0.436 (-0.040) |
AR @[ IoU=0.50:0.95 | area= all | maxDets=100 ] | 0.496 | 0.496 | 0.491 | 0.451 (-0.040) |
AR @[ IoU=0.50:0.95 | area= small | maxDets=100 ] | 0.189 | 0.193 (+0.004) | 0.284 | 0.232 (-0.052) |
AR @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] | 0.481 | 0.480 (-0.001) | 0.470 | 0.458 (-0.012) |
AR @[ IoU=0.50:0.95 | area= large | maxDets=100 ] | 0.528 | 0.528 | 0.520 | 0.462 (-0.058) |