Tokenization is the process of breaking down text into smaller units (character, word, sub-word) called tokens.

Tokenizers are used to convert the tokens into token IDs associated with them. They are also used to convert the token IDs into the tokens associated with them.

Token IDs are integers that represent the unique IDs of each token (character, word, sub-word) in the Tokenizer vocabulary (a table of all its tokens/IDs).

Example of how tokens/token IDs look in a tokenizer vocabulary:

• Tokens: large, language, model, ...
  
• Token IDs: 00001, 00002, 00003, ...
  

The words that are not in the Tokenizer vocabulary can be split into tokens (for example: tokenizers => token + izers).

The Tokenizers need to handle special cases:

• capitalization
  
• numbers
  
• languages
  
• emojis
  
• programming code
  
• ...
  

Types of tokens:

• Some tokens are complete words (for example: hello)
  
• Some tokens are parts of words (for example: token, izers)
  
• Punctuation and special characters have their own tokens
  
• Special tokens (for example: <|endoftext|>)
  
• ...
  

Each Tokenizer is characterized by its parameters:

• The size of its vocabulary
  
• The special tokens
  
• ...
  

The special tokens are unique symbols that are used to add additional information or to serve for specific purposes:

• Beginning of text
  
• End of text: used by the model to indicate that it completed processing the request
  
• Padding [PAD]: used to to align the input with the context length
  
• Unknown [UNK]: unknown token with no specific ID
  
• CLS [CLS]: used for classification tasks
  
• Masking [MASK]: used to hide a token
  
• ...
  

Example:
$ vi tokenizer.py from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("microsoft/DialoGPT-small") tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-small") input_ids = tokenizer.encode("Hello Tokenizers!", return_tensors='pt') # input tokens ids print(input_ids) # tensor([[15496, 29130, 11341, 0]]) for id in input_ids[0]: # convert the input token id to it corresponding token print(tokenizer.decode(id)) # [Hello] [ Token] [izers] [!] output = model.generate( input_ids=input_ids, max_new_tokens=50 ) # output tokens ids print(output) # tensor([[15496, 29130, 11341, 0, 50256]]) for id in output[0]: # convert the output token id to it corresponding token print(tokenizer.decode(id)) # [Hello] [ Token] [izers] [!] [<|endoftext|>] # convert the output token ids to their corresponding tokens print(tokenizer.decode(output[0])) # Hello Tokenizers!<|endoftext|>
Run the Python script:
$ python tokenizer.py
Output:
# input tokens ids tensor([[15496, 29130, 11341, 0]]) # input tokens Hello Token izers ! # output tokens ids tensor([[15496, 29130, 11341, 0, 50256]]) # output tokens Hello Token izers ! <|endoftext|> # output Hello Tokenizers!<|endoftext|>