Class: Transformers::Distilbert::DistilBertTokenizer::WordpieceTokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/transformers/models/distilbert/tokenization_distilbert.rb

Instance Method Summary collapse

Constructor Details

#initialize(vocab:, unk_token:, max_input_chars_per_word: 100) ⇒ WordpieceTokenizer

Returns a new instance of WordpieceTokenizer.



45
46
47
48
49
 
# File 'lib/transformers/models/distilbert/tokenization_distilbert.rb', line 45

def initialize(vocab:, unk_token:, max_input_chars_per_word: 100)
  @vocab = vocab
  @unk_token = unk_token
  @max_input_chars_per_word = max_input_chars_per_word
end