Module: Tokenizers::Models

Defined in:

lib/tokenizers/models/bpe.rb,
lib/tokenizers/models/unigram.rb,
lib/tokenizers/models/word_level.rb,
lib/tokenizers/models/word_piece.rb

Defined Under Namespace

Classes: BPE, Unigram, WordLevel, WordPiece