mirror of
https://github.com/opencv/opencv.git
synced 2026-09-25 04:09:57 +03:00
Jaivardhan Bhola
84c2360c27
Generalize tokenizer loading to support method-based family dispatch in DNN - #29675 Companion PR: https://github.com/opencv/opencv_extra/pull/1402 ### Changes Added ALBERT and BERT support end-to-end, with `samples/dnn/albert_inference.py `and `samples/dnn/bert_inference.py `as validation samples, plus expanded coverage in modules/dnn/test/test_tokenizer.cpp. Required changes to `cv::dnn::dnn.hpp`, `graph_fusion_attention.cpp`, and `unicode.cpp/unicode.hpp` to support Unigram and WordPiece tokenizers. To back ALBERT/BERT, generalized `cv::dnn::Tokenizer `from a single BPE implementation into a method-dispatched frontend, adding `core_wordpiece.cpp/hpp` (WordPiece) and `core_unigram.cpp/hpp` (Unigram) as new backends. `tokenizer.cpp` now routes by method across BPE, Gemma,, SentencePiece, Unigram, and WordPiece behind one shared interface. Tested against the following samples and the output matches to old tokenizer: ``` gpt2_inference.py qwen_inference.py gemma3_inference.py ``` GPT2: ``` Preparing GPT-2 model... Inferencing GPT-2 model... Hello, I'm a language model, not a programming language. I'm a language model. I'm a language model. I'm a language model. I'm a language model. I'm a ``` Gemma3: ``` Preparing Gemma3 model... Prompt: <start_of_turn>user What is OpenCV?<end_of_turn> <start_of_turn>model Inferencing Gemma3 model... Response: Okay, let's break down what OpenCV is. **What is OpenCV?** OpenCV (Open Source Computer Vision Library) is a powerful and ``` Qwen2.5: ``` Preparing Qwen2.5 model... Prompt: <|im_start|>user What is OpenCV?<|im_end|> <|im_start|>assistant Inferencing Qwen2.5 model... Response: OpenCV is a set of computer vision libraries in C++ designed to be used for image and video processing. It provides a wide range of tools and functions for ``` ### Tokenizer References Byte-level BPE: [tokenizers/src/pre_tokenizers/byte_level.rs](https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/byte_level.rs) (This defines the byte-level mapping rules, which is used in conjunction with the [BPE model](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/bpe/mod.rs)) SentencePiece BPE (Metaspace): [tokenizers/src/pre_tokenizers/metaspace.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/pre_tokenizers/metaspace.rs) (This defines the rule for replacing whitespace with the U+2581 _ character and handling byte fallback) Unigram: [tokenizers/src/models/unigram/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/unigram/mod.rs) (This contains the core logic for the Unigram lattice scoring and probabilistic tokenization rules) WordPiece: [tokenizers/src/models/wordpiece/mod.rs](https://www.google.com/search?q=https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/models/wordpiece/mod.rs) (This explicitly cites Schuster & Nakajima in the code comments and implements the greedy longest-match rule with the ## prefix) ### Pull Request Readiness Checklist - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on code under GPL or another license incompatible with OpenCV. - [x] The PR is proposed to the proper branch (`5.x`). - [x] There is a reference to the original bug report and related work. - [x] There is accuracy test and test data in `opencv_extra`, same branch name (`generalized-tokenizer`) — `bert/`, `t5/` fixtures back the new C++ tests. - [x] The feature is documented and sample code builds with project CMake.