# Section 3: Fine-Tuning - Customizing Models for Specific Tasks

## Table of Contents
1. [Introduction to Fine-Tuning](#introduction-to-fine-tuning)
2. [Why Fine-Tuning Matters](#why-fine-tuning-matters)
3. [Types of Fine-Tuning](#types-of-fine-tuning)
4. [Fine-Tuning with Microsoft Olive](#fine-tuning-with-microsoft-olive)
5. [Hands-On Examples](#hands-on-examples)
6. [Best Practices and Guidelines](#best-practices-and-guidelines)
7. [Advanced Techniques](#advanced-techniques)
8. [Evaluation and Monitoring](#evaluation-and-monitoring)
9. [Common Challenges and Solutions](#common-challenges-and-solutions)
10. [Conclusion](#conclusion)

## Introduction to Fine-Tuning

**Fine-tuning** is a powerful machine learning technique that involves adapting a pre-trained model to perform specific tasks or work with specialized datasets. Rather than training a model from scratch, fine-tuning leverages the knowledge already learned by a pre-trained model and adjusts it for your particular use case.

### What is Fine-Tuning?

Fine-tuning is a form of **transfer learning** where you:
- Start with a pre-trained model that has learned general patterns from large datasets
- Adjust the model's internal parameters using your specific dataset
- Retain the valuable knowledge while specializing the model for your task

Think of it like teaching a skilled chef to cook a new cuisine - they already understand cooking fundamentals, but need to learn specific techniques and flavors for the new style.

### Key Benefits

- **Time Efficiency**: Significantly faster than training from scratch
- **Data Efficiency**: Requires smaller datasets to achieve good performance
- **Cost-Effective**: Lower computational requirements
- **Better Performance**: Often achieves superior results compared to training from scratch
- **Resource Optimization**: Makes powerful AI accessible to smaller teams and organizations

## Why Fine-Tuning Matters

### Real-World Applications

Fine-tuning is essential in numerous scenarios:

**1. Domain Adaptation**
- Medical AI: Adapting general language models for medical terminology and clinical notes
- Legal Tech: Specializing models for legal document analysis and contract review
- Financial Services: Customizing models for financial report analysis and risk assessment

**2. Task Specialization**
- Content Generation: Fine-tuning for specific writing styles or tones
- Code Generation: Adapting models for particular programming languages or frameworks
- Translation: Improving performance for specific language pairs or technical domains

**3. Corporate Applications**
- Customer Service: Creating chatbots that understand company-specific terminology
- Internal Documentation: Building AI assistants familiar with organizational processes
- Industry-Specific Solutions: Developing models that understand sector-specific jargon and workflows

## Types of Fine-Tuning

### 1. Full Fine-Tuning (Instruction Fine-Tuning)

In full fine-tuning, all model parameters are updated during training. This approach:
- Provides maximum flexibility and performance potential
- Requires significant computational resources
- Results in a completely new version of the model
- Best for scenarios where you have substantial training data and computational resources

### 2. Parameter-Efficient Fine-Tuning (PEFT)

PEFT methods update only a small subset of parameters, making the process more efficient:

#### Low-Rank Adaptation (LoRA)
- Adds small trainable rank decomposition matrices to existing weights
- Dramatically reduces the number of trainable parameters
- Maintains performance close to full fine-tuning
- Enables easy switching between different adaptations

#### QLoRA (Quantized LoRA)
- Combines LoRA with quantization techniques
- Further reduces memory requirements
- Enables fine-tuning of larger models on consumer hardware
- Balances efficiency with performance

#### Adapters
- Insert small neural networks between existing layers
- Allow targeted fine-tuning while keeping base model frozen
- Enable modular approach to model customization

### 3. Task-Specific Fine-Tuning

Focuses on adapting models for specific downstream tasks:
- **Classification**: Adjusting models for categorization tasks
- **Generation**: Optimizing for content creation and text generation
- **Extraction**: Fine-tuning for information extraction and named entity recognition
- **Summarization**: Specializing models for document summarization

## Fine-Tuning with Microsoft Olive

Microsoft Olive is a comprehensive model optimization toolkit that simplifies the fine-tuning process while providing enterprise-grade features.

### What is Microsoft Olive?

Microsoft Olive is an open-source model optimization tool that:
- Streamlines fine-tuning workflows for various hardware targets
- Provides built-in support for popular model architectures (Llama, Phi, Qwen, Gemma)
- Offers both cloud and local deployment options
- Integrates seamlessly with Azure ML and other Microsoft AI services
- Supports automatic optimization and quantization

### Key Features

- **Hardware-Aware Optimization**: Automatically optimizes models for specific hardware (CPU, GPU, NPU)
- **Multi-Format Support**: Works with PyTorch, Hugging Face, and ONNX models
- **Automated Workflows**: Reduces manual configuration and trial-and-error
- **Enterprise Integration**: Built-in support for Azure ML and cloud deployments
- **Extensible Architecture**: Allows custom optimization techniques

### Installation and Setup

#### Basic Installation

```bash
# Create a virtual environment
python -m venv olive-env
source olive-env/bin/activate  # On Windows: olive-env\Scripts\activate

# Install Olive with auto-optimization features
pip install olive-ai[auto-opt]

# Install additional dependencies
pip install transformers onnxruntime-genai
```

#### Optional Dependencies

```bash
# For CPU optimization
pip install olive-ai[cpu]

# For GPU optimization
pip install olive-ai[gpu]

# For DirectML (Windows)
pip install olive-ai[directml]

# For Azure ML integration
pip install olive-ai[azureml]
```

#### Verify Installation

```bash
# Check Olive CLI is available
olive --help

# Verify installation
python -c "import olive; print('Olive installed successfully')"
```

## Hands-On Examples

### Example 1: Basic Fine-Tuning with Olive CLI

This example demonstrates fine-tuning a small language model for phrase classification:

#### Step 1: Prepare Your Environment

```bash
# Set up the environment
mkdir fine-tuning-project
cd fine-tuning-project

# Download sample data (optional - Olive can fetch data automatically)
huggingface-cli login  # If using private datasets
```

#### Step 2: Fine-Tune the Model

```bash
# Basic fine-tuning command
olive finetune \
  --model_name_or_path meta-llama/Llama-3.2-1B-Instruct \
  --trust_remote_code \
  --output_path models/llama/ft \
  --data_name xxyyzzz/phrase_classification \
  --text_template "<|start_header_id|>user<|end_header_id|>\n{phrase}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n{tone}" \
  --method lora \
  --max_steps 100 \
  --log_level 1
```

#### Step 3: Optimize for Deployment

```bash
# Convert to ONNX format for optimized inference
olive auto-opt \
  --model_name_or_path models/llama/ft/model \
  --adapter_path models/llama/ft/adapter \
  --device cpu \
  --provider CPUExecutionProvider \
  --use_ort_genai \
  --output_path models/llama/onnx \
  --log_level 1
```

### Example 2: Advanced Configuration with Custom Dataset

#### Step 1: Prepare Custom Dataset

Create a JSON file with your training data:

```json
[
  {
    "input": "What is machine learning?",
    "output": "Machine learning is a subset of artificial intelligence that enables computers to learn and improve from experience without being explicitly programmed."
  },
  {
    "input": "Explain neural networks",
    "output": "Neural networks are computing systems inspired by biological neural networks that learn from data through interconnected nodes or neurons."
  }
]
```

#### Step 2: Create Configuration File

```yaml
# olive-config.yaml
model:
  type: PyTorchModel
  config:
    model_path: "microsoft/DialoGPT-medium"
    task: "text-generation"

data_configs:
  - name: "custom_dataset"
    type: "HuggingfaceContainer"
    load_dataset_config:
      data_files: "path/to/your/dataset.json"
      split: "train"
    pre_process_data_config:
      text_template: "User: {input}\nAssistant: {output}"

passes:
  lora:
    type: LoRA
    config:
      r: 16
      lora_alpha: 32
      target_modules: ["c_attn", "c_proj"]
      modules_to_save: ["ln_f", "lm_head"]
```

#### Step 3: Execute Fine-Tuning

```bash
# Run with custom configuration
olive run --config olive-config.yaml --setup
```

### Example 3: QLoRA Fine-Tuning for Memory Efficiency

```bash
# Fine-tune with QLoRA for better memory efficiency
olive finetune \
  --method qlora \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --data_name nampdn-ai/tiny-codes \
  --train_split "train[:4096]" \
  --eval_split "train[4096:4224]" \
  --text_template "### Language: {programming_language} \n### Question: {prompt} \n### Answer: {response}" \
  --per_device_train_batch_size 16 \
  --per_device_eval_batch_size 16 \
  --max_steps 150 \
  --logging_steps 50 \
  --output_path adapters/tiny-codes
```

## Best Practices and Guidelines

### Data Preparation

**1. Data Quality Over Quantity**
- Prioritize high-quality, diverse examples over large volumes of poor data
- Ensure data is representative of your target use case
- Clean and preprocess data consistently

**2. Data Format and Templates**
- Use consistent formatting across all training examples
- Create clear input-output templates that match your use case
- Include appropriate instruction formatting for instruction-tuned models

**3. Dataset Splitting**
- Reserve 10-20% of data for validation
- Maintain similar distributions across train/validation splits
- Consider stratified sampling for classification tasks

### Training Configuration

**1. Learning Rate Selection**
- Start with smaller learning rates (1e-5 to 1e-4) for fine-tuning
- Use learning rate scheduling for better convergence
- Monitor loss curves to adjust rates accordingly

**2. Batch Size Optimization**
- Balance batch size with available memory
- Use gradient accumulation for larger effective batch sizes
- Consider the relationship between batch size and learning rate

**3. Training Duration**
- Monitor validation metrics to avoid overfitting
- Use early stopping when validation performance plateaus
- Save checkpoints regularly for recovery and analysis

### Model Selection

**1. Base Model Choice**
- Select models pre-trained on similar domains when possible
- Consider model size relative to your computational constraints
- Evaluate licensing requirements for commercial use

**2. Fine-Tuning Method Selection**
- Use LoRA/QLoRA for resource-constrained environments
- Choose full fine-tuning when maximum performance is critical
- Consider adapter-based approaches for multiple task scenarios

### Resource Management

**1. Hardware Optimization**
- Choose appropriate hardware for your model size and method
- Utilize GPU memory efficiently with gradient checkpointing
- Consider cloud-based solutions for larger models

**2. Memory Management**
- Use mixed precision training when available
- Implement gradient accumulation for memory constraints
- Monitor GPU memory usage throughout training

## Advanced Techniques

### Multi-Adapter Training

Train multiple adapters for different tasks while sharing the base model:

```bash
# Train multiple LoRA adapters
olive finetune --method lora --task_name "classification" --output_path adapters/classifier
olive finetune --method lora --task_name "generation" --output_path adapters/generator

# Generate multi-adapter ONNX model
olive generate-adapter \
  --base_model_path models/base \
  --adapter_paths adapters/classifier,adapters/generator \
  --output_path models/multi-adapter
```

### Hyperparameter Optimization

Implement systematic hyperparameter tuning:

```yaml
# hyperparameter-search.yaml
search_strategy:
  type: "random"
  num_trials: 20

search_space:
  learning_rate:
    type: "float"
    low: 1e-6
    high: 1e-3
    log: true
  
  lora_r:
    type: "int"
    low: 8
    high: 64
  
  batch_size:
    type: "choice"
    values: [8, 16, 32]
```

### Custom Loss Functions

Implement domain-specific loss functions:

```python
# custom_loss.py
import torch
import torch.nn as nn

class CustomContrastiveLoss(nn.Module):
    def __init__(self, margin=1.0):
        super(CustomContrastiveLoss, self).__init__()
        self.margin = margin
        
    def forward(self, output1, output2, label):
        euclidean_distance = nn.functional.pairwise_distance(output1, output2)
        loss_contrastive = torch.mean((1-label) * torch.pow(euclidean_distance, 2) +
                                    (label) * torch.pow(torch.clamp(self.margin - euclidean_distance, min=0.0), 2))
        return loss_contrastive
```

## Evaluation and Monitoring

### Metrics and Evaluation

**1. Standard Metrics**
- **Accuracy**: Overall correctness for classification tasks
- **Perplexity**: Language modeling quality measure
- **BLEU/ROUGE**: Text generation and summarization quality
- **F1 Score**: Balanced precision and recall for classification

**2. Domain-Specific Metrics**
- **Task-Specific Benchmarks**: Use established benchmarks for your domain
- **Human Evaluation**: Include human assessment for subjective tasks
- **Business Metrics**: Align with actual business objectives

**3. Evaluation Setup**

```python
# evaluation_script.py
from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_dataset
import torch

def evaluate_model(model_path, test_dataset, metric_type="accuracy"):
    """
    Evaluate fine-tuned model performance
    """
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    model = AutoModelForCausalLM.from_pretrained(model_path)
    
    # Evaluation logic here
    results = {}
    
    for example in test_dataset:
        # Process example and calculate metrics
        pass
    
    return results
```

### Monitoring Training Progress

**1. Loss Tracking**
```bash
# Enable detailed logging
olive finetune \
  --logging_steps 10 \
  --eval_steps 50 \
  --save_steps 100 \
  --logging_dir ./logs \
  --report_to tensorboard
```

**2. Validation Monitoring**
- Track validation loss alongside training loss
- Monitor for signs of overfitting (validation loss increasing while training loss decreases)
- Use early stopping based on validation metrics

**3. Resource Monitoring**
- Monitor GPU/CPU utilization
- Track memory usage patterns
- Monitor training speed and throughput

## Common Challenges and Solutions

### Challenge 1: Overfitting

**Symptoms:**
- Training loss continues to decrease while validation loss increases
- Large gap between training and validation performance
- Poor generalization to new data

**Solutions:**
```yaml
# Regularization techniques
passes:
  lora:
    type: LoRA
    config:
      r: 16  # Reduce rank to prevent overfitting
      lora_alpha: 16  # Lower alpha value
      lora_dropout: 0.1  # Add dropout
      weight_decay: 0.01  # L2 regularization
```

### Challenge 2: Memory Limitations

**Solutions:**
- Use gradient checkpointing
- Implement gradient accumulation
- Choose parameter-efficient methods (LoRA, QLoRA)
- Utilize model parallelism for large models

```bash
# Memory-efficient training
olive finetune \
  --method qlora \
  --gradient_checkpointing true \
  --per_device_train_batch_size 1 \
  --gradient_accumulation_steps 16
```

### Challenge 3: Slow Training

**Solutions:**
- Optimize data loading pipelines
- Use mixed precision training
- Implement efficient batching strategies
- Consider distributed training for large datasets

```yaml
# Performance optimization
training_config:
  fp16: true  # Mixed precision
  dataloader_num_workers: 4
  optim: "adamw_torch"
  lr_scheduler_type: "cosine"
```

### Challenge 4: Poor Performance

**Diagnosis Steps:**
1. Verify data quality and formatting
2. Check learning rate and training duration
3. Evaluate base model choice
4. Review preprocessing and tokenization

**Solutions:**
- Increase training data diversity
- Adjust learning rate schedule
- Try different base models
- Implement data augmentation techniques

## Conclusion

Fine-tuning is a powerful technique that democratizes access to state-of-the-art AI capabilities. By leveraging tools like Microsoft Olive, organizations can efficiently adapt pre-trained models to their specific needs while optimizing for performance and resource constraints.

### Key Takeaways

1. **Choose the Right Approach**: Select fine-tuning methods based on your computational resources and performance requirements
2. **Data Quality Matters**: Invest in high-quality, representative training data
3. **Monitor and Iterate**: Continuously evaluate and improve your models
4. **Leverage Tools**: Use frameworks like Olive to simplify and optimize the process
5. **Consider Deployment**: Plan for model optimization and deployment from the beginning


## ➡️ What's next

- [04: Deployment - Production-Ready Model Implementation](./04.SLMOps.Deployment.md)