To save a model, tokenizer, and configuration files, we can use the "
save_pretrained" method from the Hugging Face Transformers library.
Ideally, you will save all related files in the same folder.
Note that saving the model also saves its configuration file.
-
Save the model and its associated configuration files:
Python code:
Run the Python script:
This will create a directory containing:
-
Save the model tokenizer files:
Python code:
Run the Python script:
This will create a directory containing:
-
Save only the model configuration file:
Python code:
Run the Python script:
This will create a directory containing:
Files:
-
config.json: The configuration file of the model.
-
tokenizer_config.json: The configuration file of the tokenizer.
-
vocab.json, tokenizer.json: contain the vocabulary and the mapping of tokens to IDs.
-
special_tokens_map.json: contains the mapping of special tokens used by the tokenizer.
-
model.safetensors: contains the model's weights.
-
generation_config.json, merges.txt