subwords

🛠 Better command line editing tools for Zsh

Repository of "An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction" (EMNLP-IJCNLP 2019)

High-precision, zero-dependency Byte-Fallback Unigram tokenizer with dual-offset tracking, arithmetic isolation, and multilingual Unicode protection.

Evaluation for unsupervised morphological analysis and segmentation

Classifying English, Slovak, Czech language using Naive Bayes

The concept of DAWGs is based on: Blumer, A. et al. (1985). The smallest automation recognizing the subwords of a text. Theoretical Computer Science, 40, 31–55.

Classified sentences into one of Slovak, Czech, and English. Implemented relevant preprocessing steps, addressed the class imbalance in training set by employing the learned theory of Naive Bayes Models, and implementing subword units.

Subword tokenization methods (BPE, SentencePiece, WordPiece, Unigram) in Julia for text ('keemena') preprocessing pipelines.