Class: Informers::BertTokenizer
- Inherits:
-
PreTrainedTokenizer
- Object
- PreTrainedTokenizer
- Informers::BertTokenizer
- Defined in:
- lib/informers/tokenizers.rb
Instance Attribute Summary
Attributes inherited from PreTrainedTokenizer
#mask_token, #mask_token_id, #sep_token_id
Method Summary
Methods inherited from PreTrainedTokenizer
#batch_decode, #call, #convert_tokens_to_ids, #convert_tokens_to_string, #decode, #get_token, #id_to_token, #initialize, #padding_side=
Constructor Details
This class inherits a constructor from Informers::PreTrainedTokenizer