Class: Informers::DistilBertTokenizer

Inherits:
PreTrainedTokenizer show all
Defined in:
lib/informers/tokenizers.rb

Instance Attribute Summary

Attributes inherited from PreTrainedTokenizer

#mask_token, #mask_token_id, #sep_token_id

Method Summary

Methods inherited from PreTrainedTokenizer

#batch_decode, #call, #convert_tokens_to_ids, #convert_tokens_to_string, #decode, #get_token, #id_to_token, #initialize, #padding_side=

Constructor Details

This class inherits a constructor from Informers::PreTrainedTokenizer