LLMs | Introduction
  1. Introduction
  2. Representation Models (Encoder-Only)
  3. Generative Models (Decoder-Only)
  4. Encoder-Decoder Models
  5. Creating a language model
  1. Introduction
    Large Language Models (LLMs) are a class of AI models designed to understand, generate, and interact with human language. These models are built to interact with language in human-like ways (e.g., answering questions, writing essays, holding conversations). Built on the Transformer architecture, LLMs are trained on massive text corpora and learn to capture complex linguistic patterns, semantics, and contextual relationships within language. These models consist of neural networks with billions of parameters — numeric values that capture the model's understanding of language. During training, these parameters are adjusted to optimize performance on language-specific tasks.

    Types of LLMs:
    • Representation Models (Encoder-Only)
    • Generative Models (Decoder-Only)
    • Encoder-Decoder Models

    Applications of LLMs:
    • Text generation: Creative writing, content generation.
    • Text classification: Spam detection, sentiment analysis.
    • Text clustering: Organizing unstructured data.
    • Semantic search: Context-aware information retrieval.

    In this tutorial, I will focus only on CPU-based LLM development, which is well-suited for learning and experimentation.
  2. Representation Models (Encoder-Only)
    • Function: Designed for understanding and encoding language rather than generating it.
    • Architecture: Encoder-only Transformer architecture.
    • Model type: Sequence-to-value (sometimes sequence-to-sequence for encoding tasks like translation).
    • Use case: Takes an input sequence and produces a classification, embedding, or other representations.

    Example:
    Input: “The weather today is great!”
Output: 0.95 (positive sentiment score)
    These models are commonly used in:
    • Text classification and sentiment analysis
    • Named entity recognition (NER)
    • Text embeddings

    Examples of representation models:
    • BERT (Bidirectional Encoder Representations from Transformers) – Open-source: 340 million parameters
    • RoBERTa (Robustly Optimized BERT Pretraining Approach) - Open-source: 355 million parameters
    • DeBERTa (Decoding-enhanced BERT with Disentangled Attention) - Open-source: 1024 million parameters
  3. Generative Models (Decoder-Only)
    • Function: Focused on generating coherent and contextually relevant text.
    • Architecture: Decoder-only Transformer architecture.
    • Model type: Sequence-to-sequence.
    • Use case: Takes a text input (prompt) and generates a text output (completion).

    Example:
    Input: “What’s 1 + 1?”
Output: “The answer is 2.”
    These models are prompt-driven and require clear instructions to produce useful responses. They are not typically task-specific but can be fine-tuned for specific applications.

    These models are commonly used in:
    • Translation
    • Question answering
    • Summarization

    Examples of generative models:
    • GPT-4 (Generative Pre-trained Transformer) - OpenAI - Proprietary: 1.75 trillion parameters

    • LLaMA 4 (Large Language Model Meta AI) - Meta AI - Open-source: 2 trillion parameters

    • Gemini - Google - Proprietary: 27 billion parameters
    • Claude - Anthropic - Proprietary
  4. Encoder-Decoder Models
    • Function: Combine the strengths of both understanding (encoding) and generation (decoding).
    • Architecture: Full Transformer (encoder + decoder stacks).
    • Model type: Sequence-to-sequence.
    • Use case: Effective for tasks that need both comprehension and generation, like translation, summarization, and question answering.

    Example: Masked language modeling
    Input: “LLMs can be used for [MASK], a form of [MASK].”
Output:
[MASK] = text generation
[MASK] = generative AI
    These models are commonly used in:
    • Translation
    • Question answering
    • Summarization

    Examples of encoder-decoder models:
    • T5 (Text-to-Text Transfer Transformer) – Google: 11 billion parameters
    • BART (Bidirectional and Auto-Regressive Transformers) – Facebook
  5. Creating a language model
    • Training (Pre-training)
      This is the initial phase where a model learns the structure and patterns of language from a large, diverse dataset.

      Characteristics:
      • Produces foundation models (also called base models).
      • Learns syntax, semantics, and general knowledge.
      • Focuses on next-token prediction (for generative models).
      • Utilizes unsupervised learning techniques.
      • Requires extensive computational resources (GPUs, TPUs, large memory).
      • Requires vast amounts of data.
      • Requires significant training time.

    • Fine-tuning
      This step customizes the pre-trained model for specific tasks or domains.

      Characteristics:
      • Produces fine-tuned models.
      • Involves supervised learning with labeled datasets.
      • Trains on smaller datasets and consumes fewer resources.
      • Shorter training times compared to pre-training.
      • Enables models to follow instructions.

    Training Techniques
    • Supervised Learning: Uses labeled data for tasks like classification and regression.
      Example: Sentiment analysis where text is labeled as positive or negative.

    • Unsupervised Learning: Uses unlabeled data for discovering patterns.
      Example: Clustering similar documents or learning embeddings.

    • Masked Language Modeling (MLM)
      • Key technique for training representation models like BERT.
      • Randomly masks tokens in input text.
      • The model learns to predict the masked tokens based on context.

      Example:
      Input: “[CLS] LLMs are large [MASK] models.”
Output: language
© 2025  mtitek