Embeddings are numerical representations of a token:

• They are also known as embedding vectors, vector representations, or vectors.
  
• They aim to capture the semantic (meanings, context, patterns) of the embedded text.
  
• They represent a token and its links with other tokens based on the contexts (contextualized tokens embeddings)
  
• They allow measuring the semantic similarity between two tokens.
  

Embeddings are used for:

• Text generation
  
• Text classification
  
• Text clustering
  
• ...
  

Embeddings have the following characteristics:

• The size (number of dimensions) of the embeddings is fixed and depends on the underlying embedding model.
  
• Each cell in the dimension row has a numerical value (property).
  
• The properties (numerical values) of the embeddings are used to represent tokens.
  

A tokenizer is used in the training process of its associated model:

• The model is linked with its tokenizer and can't be used with another tokenizer.
  
• The model holds an embedding for each token in the tokenizer's vocabulary.
  
• When a model is created, its embeddings and weights are randomly initialized.
  
• When a model is trained, its embeddings and weights are assigned proper values that capture the semantic of each token.
  

Types of embeddings:

• Tokens embeddings: creating one vector that represents each token.
  
• Sentences embeddings: creating one vector that represents each sentence (used for categorization, semantic search RAG).
  
• Documents embeddings: creating one vector that represents each document (used for categorization, semantic search RAG).
  

When generating embeddings for text (longer than just one token) the embeddings should capture the meaning of the whole text. One way to generate those embeddings is to average the values of all the tokens' embeddings of the text.

Download the model:
$ huggingface-cli download microsoft/deberta-v3-xsmall
Python code:
$ vi embeddings.py from transformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained("microsoft/deberta-v3-xsmall") tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base") tokens = tokenizer('Hello Embeddings!', return_tensors='pt') for token in tokens['input_ids'][0]: # convert the input token id to it corresponding token print(tokenizer.decode(token)) # [CLS] [Hello] [Emb] [edd] [ings] [!] [SEP] output = model(**tokens)[0] # output shape: [batch_size, number_of_tokens, embeddings_dimension] print(output.shape) # torch.Size([1, 7, 384]) # output embeddings print(output)
Run the Python script:
$ python embeddings.py
Output:
# input tokens [CLS] Hello Emb edd ings ! [SEP] # output shape: [batch_size, number_of_tokens, embeddings_dimension] torch.Size([1, 7, 384]) # output embeddings tensor([[[-3.3186, 0.1003, -0.1506, ..., -0.2840, -0.3882, -0.1670], [-0.5446, 0.7986, -0.4200, ..., 0.1163, -0.3322, -0.3622], [-0.1689, 0.6443, -0.0145, ..., 0.0207, -0.5754, 1.3607], ..., [ 0.0366, 0.0818, -0.0607, ..., -0.4793, -0.7831, -0.9185], [-0.0555, 0.3136, 0.2662, ..., 0.3092, -0.4876, -0.3294], [-3.1255, 0.1324, -0.0899, ..., -0.1426, -0.5295, 0.0731]]], grad_fn=<NativeLayerNormBackward0>)
Note that the created embeddings have the follwong size: 1, 7, 384

• 1: the batch dimenesion
  
• 7: seven tokens
  
• 384: each token is embedded in a vector of 384 values
  

The batch dimension can be larger than 1 in cases when multiple sentences are given to the model to be processed at the same time.

Install the Sentence Transformers library:
$ pip install sentence-transformers
Python code:
$ vi embeddings.py from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") sentences = ["Hello Sentence Transformers library!.", "Generate sentence embeddings!"] embeddings = model.encode(sentences) # output shape: [number_of_tokens, embeddings_dimension] print(embeddings.shape) # (2, 384) # output embeddings print(embeddings)
Run the Python script:
$ python embeddings.py
Output:
# output shape: [number_of_tokens, embeddings_dimension] (2, 384) # output embeddings [[-6.70545474e-02 -3.04300548e-03 3.52957926e-04 4.17553373e-02 5.08048979e-04 1.49061205e-02 1.29256323e-02 5.43267690e-02 ... 1.12174535e-02 1.13273829e-01 5.92597015e-02 -1.89474523e-02]]