byte-fallback

Here is 1 public repository matching this topic...

High-precision, zero-dependency Byte-Fallback Unigram tokenizer with dual-offset tracking, arithmetic isolation, and multilingual Unicode protection.

python nlp rust viterbi-algorithm deep-learning tokenizer hacktoberfest tokenization unigram good-first-issue pyo3 subwords byte-fallback

To associate your repository with the byte-fallback topic, visit your repo's landing page and select "manage topics."