unigram

Telegram for Windows

Build unigram and bigram language models, implement Laplace smoothing and use the models to compute the perplexity of test corpora.

Get n-grams from text

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Next Word Prediction using n-gram Probabilistic Model with various Smoothing Techniques

The Unigram website build with React and Bootstrap.

NLP tutorials and guidelines to learn efficiently

High-precision, zero-dependency Byte-Fallback Unigram tokenizer with dual-offset tracking, arithmetic isolation, and multilingual Unicode protection.

Telegram bot that generates random messages using a Markov chain, now for Node.js

Unigram Tokenization realization from scratch

All AUT's principles and applications of artificial intelligence course projects.

High-Performance Azerbaijani Tokenizers (30% fewer tokens, 40% faster than multilingual alternatives)

NLP extra project at AUT Artificial Intelligence course (Fall 2020)

Word2Vec using Hierarchy Softmax and Negative Sampling with Unigram & Subsampling

Python Web Crawler implementing Iterative Deepening Depth Search

UNB Fall-2018 NLP Assignments 💬

We have implemened an NLP Project to recognize the correct poets. In the project, we have used the bigram, unigram and backoff model for smoothing.

Performance evaluation of sentiment classification on movie reviews

COMP6721: Introduction to Artificial Intelligence Assignments (GOFAI, Machine Learning, NLP)