Libraries
»
tokenizers
(0.5.3)
»
Index (M)
»
Tokenizers
»
Models
Module: Tokenizers::Models
Defined in:
lib/tokenizers/models/bpe.rb
,
lib/tokenizers/models/unigram.rb,
lib/tokenizers/models/word_level.rb,
lib/tokenizers/models/word_piece.rb
Defined Under Namespace
Classes:
BPE
,
Unigram
,
WordLevel
,
WordPiece