Class: Transformers::Bert::BertTokenizer::WordpieceTokenizer

Inherits:
Object
  • Object
show all
Defined in:
lib/transformers/models/bert/tokenization_bert.rb

Instance Method Summary collapse

Constructor Details

#initialize(vocab:, unk_token:, max_input_chars_per_word: 100) ⇒ WordpieceTokenizer

Returns a new instance of WordpieceTokenizer.



40
41
42
43
44
 
# File 'lib/transformers/models/bert/tokenization_bert.rb', line 40

def initialize(vocab:, unk_token:, max_input_chars_per_word: 100)
  @vocab = vocab
  @unk_token = unk_token
  @max_input_chars_per_word = max_input_chars_per_word
end