Fine-Tuning Methodologies

Source Excerpt

Overview of Large Language Model fine-tuning approaches — from supervised fine-tuning (SFT) to reinforcement learning from human feedback (RLHF) and beyond. Each methodology optimizes different objectives and requires different data formats.

Supervised Fine-Tuning (SFT)

Description

The simplest and most common fine-tuning method. Train the model on labeled input-output pairs using standard cross-entropy loss.

Data Format

{
  "instruction": "Summarize this article.",
  "input": "Article text...",
  "output": "Summary..."
}

Or conversational format:

{
  "conversations": [
    {"from": "human", "value": "Summarize this article.\n\nArticle text..."},
    {"from": "gpt", "value": "Summary..."}
  ]
}

Loss Function

L_SFT = -Σ log P(y_i | x, y_<i)

Standard next-token prediction loss on the output tokens only.

Strengths

• Simple to implement
• Works well for most tasks
• Large dataset availability
• Fast training

Weaknesses

• Doesn't optimize for preference/quality
• May produce verbose or low-quality outputs
• No alignment with human values

Direct Preference Optimization (DPO)

Description

Optimize the model directly on preference data (chosen vs rejected responses) without a separate reward model. Simplifies RLHF by combining reward modeling and policy optimization into a single step.

Data Format

{
  "prompt": "Write a Python function to sort a list.",
  "chosen": "def sort_list(lst):\n    return sorted(lst)",
  "rejected": "Here's how you might sort a list in Python..."
}

Loss Function

L_DPO = -log σ(β log π(chosen/prompt) - β log π(rejected/prompt))

Where π is the policy model, σ is sigmoid, and β controls strength.

Strengths

• No separate reward model needed
• More stable than RLHF
• Simpler pipeline
• Good alignment results

Weaknesses

• Requires preference data (harder to collect than SFT data)
• May over-optimize on preference signal
• β hyperparameter is sensitive

Online Reinforcement Learning from Human Feedback (RLHF)

Description

The classic alignment pipeline: train a reward model on human preferences, then optimize the policy using PPO (Proximal Policy Optimization).

Pipeline

SFT → Reward Model Training → PPO Fine-Tuning → Aligned Model

Data Format

Same as DPO (preference pairs), but used to train a separate reward model first.

Strengths

• State-of-the-art alignment quality
• Flexible reward specification
• Well-understood methodology

Weaknesses

• Complex pipeline (3 stages)
• PPO is unstable and hyperparameter-sensitive
• Requires significant compute
• Reward hacking risk

Group Relative Policy Optimization (GRPO)

Description

A variant of RLHF that uses group-relative advantages instead of absolute rewards. More stable than PPO, less prone to reward hacking.

How It Works

1. Generate multiple responses per prompt
2. Score all responses (reward model or verifier)
3. Compute relative advantages within the group
4. Update policy using relative advantages

Strengths

• More stable than PPO
• Less reward hacking
• No separate value model needed
• Good for jailbreak resistance

Weaknesses

• Requires multiple generations per prompt (slower)
• Hyperparameter tuning needed

Odds Ratio Preference Optimization (ORPO)

Description

Optimizes the odds ratio between chosen and rejected responses. Simpler than DPO, more stable in practice.

Loss Function

L_ORPO = -log(π(chosen) / (π(chosen) + π(rejected)))

Strengths

• Simpler than DPO
• More stable training
• Good results with less data

Weaknesses

• Less theoretically grounded than DPO
• Fewer implementations

Kahneman-Tversky Optimization (KTO)

Description

Inspired by prospect theory. Optimizes based on desired/undesired labels rather than pairwise preferences. More flexible data format.

Data Format

{
  "prompt": "Write a Python function.",
  "completion": "def sort_list(lst): return sorted(lst)",
  "label": true  # true = desired, false = undesired
}

Strengths

• Flexible data format (no pairing needed)
• Works with implicit feedback
• Good for diverse preference signals

Weaknesses

• Less intuitive than DPO
• Fewer implementations

Method Selection Guide

Training Pipeline

Typical Production Pipeline

1. SFT (instruction tuning) → Base capable model
2. DPO (preference optimization) → Aligned model
3. GRPO (jailbreak resistance) → Safe model

Data Requirements

Source excerpt truncated at 220 of 229 lines. Open the canonical wiki path above for the full page.