A stakeholder wants to fine-tune an LLM on their company documentation so it 'knows' their product. What do you tell them?

That fine-tuning is the wrong tool for that goal. Fine-tuning adjusts behaviour, format and style; it is an unreliable and expensive way to inject facts, and the facts go stale the moment the docs change. RAG is the correct approach for knowledge. I would propose RAG first, then consider fine-tuning only if the remaining gap is about tone, output format, or a domain-specific task the base model performs poorly. Goal | Right tool | Why. Model should know current facts | RAG | Updatable instantly, citable, no retraining. Consistent output format/schema | Fine-tuning or constrained decoding | Behaviour, not knowledge. Specific tone or brand voice | Fine-tuning (often LoRA) | Style is exactly what weights encode well. Domain jargon and task patterns | Fine-tuning | Base model genuinely lacks the pattern. Lower latency/cost at fixed quality | Fine-tune a small model on large-model outputs | Distillation — a strong, underused play. Model refuses valid domain requests | Fine-tuning on domain examples | Adjusts the behavioural prior Why fine-tuning for facts fails: Facts learned in fine-tuning are stored diffusely across weights with no provenance. The model cannot cite a source, cannot tell you when it is unsure, will confidently interpolate between facts, and needs a full retrain when a document changes. Recent work also shows fine-tuning on new facts increases hallucination — the model learns the confident style of asserting facts faster than it learns the facts themselves. How to run the conversation Establish the actual failure: “Show me 20 outputs that are wrong.” Nine times out of ten the failures are retrieval failures or prompt failures, not capability failures. This single request reframes the whole project.. Set up an evaluation before anything else: Without a sco

Explain LoRA mathematically. What does QLoRA add, and what rank would you choose?

LoRA freezes the pre-trained weight W and learns a low-rank update ΔW = BA where B is d×r and A is r×k with r ≪ d. You train ~0.1–1% of the parameters, so optimiser state collapses and adapters are megabytes rather than gigabytes. QLoRA adds 4-bit NF4 quantisation of the frozen base weights plus double quantisation and paged optimisers, letting you fine-tune a 65B model on a single 48GB GPU. Rank 8–16 covers most style/format tasks; 64–128 for genuinely new capabilities. h = W_0x + W x = W_0 x + rBAx, B R^d r, A R^r k Two initialisation details matter. A is random-normal and B is zero, so W = 0 at step one and training begins exactly at the pre-trained model. And the /r scaling means changing rank does not require re-tuning the learning rate — is usually set to 2r or fixed at 16–32. The hypothesis behind LoRA: Fine-tuning updates have low intrinsic rank. Adapting a model to a new task does not require arbitrary changes to a 4096×4096 matrix — it requires a change that lives in a much smaller subspace. Empirically rank 8 recovers most of full fine-tuning quality on style and format tasks, which is strong evidence the hypothesis holds. What QLoRA adds NF4 quantisation — a 4-bit datatype that is information-theoretically optimal for normally-distributed weights (which pre-trained weights approximately are). The frozen base drops from 14 GB to ~3.5 GB for a 7B model.. Double quantisation — quantise the quantisation constants themselves, saving another ~0.4 bits per parameter.. Paged optimisers — page optimiser state to CPU RAM on memory spikes, so a long-sequence batch does not OOM the run.. Compute stays in bf16. Weights are stored in 4 bits and dequantised on the fly for each matmul. You save memory, not compute — QLoRA is typically ~30% slower per step than LoRA. Me

Explain RLHF and how DPO differs. Why did DPO become popular?

RLHF has three stages: supervised fine-tuning, training a reward model on human preference pairs, then optimising the policy against that reward with PPO plus a KL penalty to the reference model. DPO proves the optimal RLHF policy has a closed form, which lets you skip the reward model and the RL loop entirely and optimise a simple classification-style loss directly on preference pairs. DPO won on simplicity, stability and cost — it needs two models in memory instead of four. RLHF, stage by stage 1 — Supervised fine-tuning (SFT): Train on high-quality demonstration data (prompt → ideal response). This establishes format and basic helpfulness, and gives the reference policy π_ref.. 2 — Reward model: Collect pairs (chosen, rejected) from human labellers and train a scalar reward head with the Bradley–Terry loss - (r_ (x,y_w) - r_ (x,y_l)). Only relative preference is learned — the absolute scale is arbitrary.. 3 — PPO against the reward model: Maximise E[r_ (x,y)] - , KL( _ | _ref). The KL term is essential: without it the policy finds adversarial inputs that score highly under the reward model while being nonsense — reward hacking. Why RLHF is operationally painful: PPO training keeps four models in memory: policy, reference, reward model and value/critic head. It is sensitive to a dozen hyperparameters (KL coefficient, clip range, GAE λ, value loss weight, rollout length), and failure often looks like plausible-but-degraded output rather than an obvious crash. Reproducing published RLHF results was notoriously difficult, which is precisely the gap DPO filled. The DPO derivation, in one move The KL-constrained reward maximisation has a known closed-form optimum: ^*(y x) = 1Z(x) _ref(y x) ! ( 1 r(x,y) ) Rearranging gives the reward implicitly in terms of the policy:

After fine-tuning on your domain data, the model gets worse at general tasks. What is happening and how do you fix it?

Catastrophic forgetting: gradient descent on a narrow distribution moves weights away from the configuration that encoded general capability, and nothing in the loss penalises that. Fixes ranked by effectiveness: mix 5–30% general instruction data into the fine-tuning set, use LoRA so the base weights are frozen, lower the learning rate and train fewer epochs, and always evaluate on a general benchmark alongside your domain metric. The mechanism is straightforward. Your loss only measures domain performance. Any weight configuration that minimises it is acceptable to the optimiser, including ones that destroy unrelated capabilities. There is no term in the objective that says 'and stay good at everything else'. Mitigation | Mechanism | Effectiveness. Mix in general data (15–30%) | Keeps the general distribution in the loss | Highest — do this first. LoRA instead of full FT | Base weights frozen; low-rank update is a bounded perturbation | High, and free. Lower LR + fewer epochs | Smaller total displacement from the base weights | High; 1–3 epochs is usually enough. Early stopping on a general benchmark | Stops before the damage accumulates | High, cheap to add. KL penalty to the base model | Explicitly penalises drift in output distribution | Effective but adds a second forward pass. Model merging / weight averaging | Interpolate fine-tuned and base weights | Surprisingly effective (model soups). Elastic weight consolidation | Penalise changes to high-Fisher-information weights | Theoretically clean, rarely used at LLM scale Model merging is underrated: Simply averaging the fine-tuned and base weights, = _ft + (1- ) _base, often recovers most general capability while retaining most domain gains — with zero additional training. Sweep λ over 0.3–0.9 and pick on your com

How would you build a fine-tuning dataset? How much data do you need and what makes an example good?

Quality and diversity beat volume decisively. LIMA showed 1,000 carefully curated examples can outperform 50,000 noisy ones for instruction tuning. Target 1,000–10,000 examples for style and format tasks, 10,000–100,000 for genuinely new capability. Good examples are diverse in phrasing and task type, consistent in output format, contain no contradictions, and — critically — no example should require knowledge the model does not have. The LIMA result to cite: 'Less Is More for Alignment' fine-tuned Llama-65B on 1,000 hand-curated examples and matched or beat models trained on 52,000 auto-generated ones in human evaluation. The interpretation: SFT mostly teaches format and register, not knowledge — and format is learnable from a small, consistent, diverse set. Citing this reframes the whole data conversation from volume to curation. Goal | Examples needed | Notes. Output format / schema | 500–2,000 | Highly consistent formatting matters more than count. Tone and brand voice | 1,000–5,000 | Diverse topics, consistent register. Domain task adaptation | 5,000–50,000 | Needs coverage of the real task distribution. New capability | 50,000+ | Often better solved by a different base model. Preference tuning (DPO) | 2,000–20,000 pairs | Pair quality dominates; hard pairs teach most Where the data comes from Mine production logs: The highest-value source: real user requests with outcomes you can grade. Filter for interactions a human confirmed as good. This automatically matches your real distribution — which synthetic data never quite does.. Frontier-model distillation: Generate responses with a strong model, then filter with a rubric-based judge and human spot-checks. Cheap and effective. Check the provider's terms — some prohibit training competing models on their outputs..

LoRA, QLoRA & RLHF vs DPO — LLM Fine-tuning Interview Questions

Intermediatefine-tuningragdecision-makingAsked at Anthropic, OpenAI solutions

30-second answer

That fine-tuning is the wrong tool for that goal. Fine-tuning adjusts behaviour, format and style; it is an unreliable and expensive way to inject facts, and the facts go stale the moment the docs change. RAG is the correct approach for knowledge. I would propose RAG first, then consider fine-tuning only if the remaining gap is about tone, output format, or a domain-specific task the base model performs poorly.

Goal	Right tool	Why
Model should know current facts	RAG	Updatable instantly, citable, no retraining
Consistent output format/schema	Fine-tuning or constrained decoding	Behaviour, not knowledge
Specific tone or brand voice	Fine-tuning (often LoRA)	Style is exactly what weights encode well
Domain jargon and task patterns	Fine-tuning	Base model genuinely lacks the pattern
Lower latency/cost at fixed quality	Fine-tune a small model on large-model outputs	Distillation — a strong, underused play
Model refuses valid domain requests	Fine-tuning on domain examples	Adjusts the behavioural prior

How to run the conversation

Establish the actual failure
“Show me 20 outputs that are wrong.” Nine times out of ten the failures are retrieval failures or prompt failures, not capability failures. This single request reframes the whole project.
Set up an evaluation before anything else
Without a scored eval set you cannot tell whether fine-tuning helped. Building 100–200 graded examples is a day of work and it is the highest-leverage day in the project.
Exhaust the cheap levers
Better system prompt, few-shot examples, structured output constraints, a reranker on the retrieval step. Each is hours, not weeks, and reversible.
Then propose fine-tuning with a scope
“If after RAG we still see formatting inconsistency and the wrong register, I'd LoRA-tune a 7B on ~2,000 curated examples. That is roughly a day of GPU time and I can A/B it against the RAG-only baseline.” Concrete scope, concrete cost, measurable outcome.

pythonThe cost comparison that usually ends the debate

def compare(monthly_requests, in_tok=2000, out_tok=500):
    # Frontier API
    api = monthly_requests * (in_tok * 3e-6 + out_tok * 15e-6)

    # RAG on the same frontier model: more input tokens, no training cost
    rag = monthly_requests * ((in_tok + 3000) * 3e-6 + out_tok * 15e-6) + 200  # + infra

    # Fine-tuned 8B self-hosted: one-off training + GPU hours
    train_once = 300                          # ~1 A100-day, LoRA on 8B
    gpu_month  = 1.80 * 24 * 30               # one A10G reserved
    ft = gpu_month + (train_once / 12)        # amortise training over a year

    return api, rag, ft

for n in (10_000, 500_000, 10_000_000):
    api, rag, ft = compare(n)
    print(f"{n:>10,} req/mo   API ${api:>10,.0f}   RAG ${rag:>10,.0f}   FT-8B ${ft:>9,.0f}")

#     10,000 req/mo   API $       135   RAG $       425   FT-8B $    1,321
#    500,000 req/mo   API $     6,750   RAG $    15,200   FT-8B $    1,321
# 10,000,000 req/mo   API $   135,000   RAG $  300,200   FT-8B $    1,321
#
# Below ~50k requests/month the API wins outright. Above ~500k a fine-tuned
# small model is 5-100x cheaper. Volume, not sophistication, decides this.

Say these points

Fine-tuning teaches behaviour and style; RAG supplies knowledge.
Fine-tuned facts have no provenance, go stale, and can increase hallucination.
Order: prompt → few-shot → RAG → fine-tune. Each step is cheaper to iterate.
Build the eval set before any intervention or you cannot measure anything.
The best fine-tuning use case is distilling a frontier model into a small one for cost.

Avoid these mistakes

Agreeing to fine-tune for knowledge injection without pushback.
Starting a fine-tune with no evaluation set.
Ignoring that self-hosting adds ops burden the team may not want.

Expect these follow-ups

How would you build an eval set for a domain assistant?
When would you fine-tune *and* use RAG together?
How do you decide the size of the fine-tuning dataset?

AdvancedloraqlorapeftAsked at Hugging Face, Together AI

30-second answer

LoRA freezes the pre-trained weight W and learns a low-rank update ΔW = BA where B is d×r and A is r×k with r ≪ d. You train ~0.1–1% of the parameters, so optimiser state collapses and adapters are megabytes rather than gigabytes. QLoRA adds 4-bit NF4 quantisation of the frozen base weights plus double quantisation and paged optimisers, letting you fine-tune a 65B model on a single 48GB GPU. Rank 8–16 covers most style/format tasks; 64–128 for genuinely new capabilities.

h = W_{0} x + Δ W x = W_{0} x + \frac{α}{r} B A x, B \in R^{d \times r}, A \in R^{r \times k}

Two initialisation details matter. A is random-normal and B is zero, so $Δ W = 0$ at step one and training begins exactly at the pre-trained model. And the $α / r$ scaling means changing rank does not require re-tuning the learning rate — $α$ is usually set to $2 r$ or fixed at 16–32.

pythonLoRA from scratch, plus the parameter accounting

import torch, torch.nn as nn, math

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.05):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False                       # freeze W0

        self.r, self.scaling = r, alpha / r
        self.lora_A = nn.Parameter(torch.empty(r, base.in_features))
        self.lora_B = nn.Parameter(torch.zeros(base.out_features, r))   # ZERO
        nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.dropout(x) @ self.lora_A.T @ self.lora_B.T * self.scaling

    @torch.no_grad()
    def merge(self):
        """Fold the adapter into W0 for zero-overhead inference."""
        self.base.weight += (self.lora_B @ self.lora_A) * self.scaling
        return self.base

# Accounting for a 7B model, LoRA r=16 on attention projections only
d, layers, r = 4096, 32, 16
lora_params = layers * 4 * 2 * r * d          # 4 matrices, A and B each
print(f"trainable: {lora_params/1e6:.1f}M  ({100*lora_params/7e9:.3f}% of the model)")
print(f"adapter file: {lora_params*2/1e6:.0f} MB in bf16")
# trainable: 16.8M  (0.240% of the model)
# adapter file: 34 MB in bf16
#
# Memory: full FT of 7B needs ~112GB. LoRA needs ~14GB (frozen bf16 weights)
# + ~0.2GB optimizer state -> fits on a single 24GB consumer GPU.

What QLoRA adds

NF4 quantisation — a 4-bit datatype that is information-theoretically optimal for normally-distributed weights (which pre-trained weights approximately are). The frozen base drops from 14 GB to ~3.5 GB for a 7B model.
Double quantisation — quantise the quantisation constants themselves, saving another ~0.4 bits per parameter.
Paged optimisers — page optimiser state to CPU RAM on memory spikes, so a long-sequence batch does not OOM the run.
Compute stays in bf16. Weights are stored in 4 bits and dequantised on the fly for each matmul. You save memory, not compute — QLoRA is typically ~30% slower per step than LoRA.

Method	Trainable	Memory (7B)	Quality vs full FT
Full fine-tuning	100%	~112 GB	Baseline
LoRA r=16	0.24%	~16 GB	~99% on style/format tasks
QLoRA r=64	1.0%	~6 GB	~99% (paper's headline result)
Prefix/prompt tuning	<0.1%	~15 GB	Weaker; sensitive to length
(IA)³	~0.01%	~15 GB	Good for few-shot adaptation

pythonProduction QLoRA configuration

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",                    # NOT "fp4" -- nf4 is better
    bnb_4bit_compute_dtype=torch.bfloat16,        # matmuls in bf16
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B", quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)

lora = LoraConfig(
    r=64, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
    # Target ALL linear layers, not just q_proj/v_proj -- the QLoRA paper found
    # this matters more than increasing rank.
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()
# trainable params: 167,772,160 || all params: 8,197,878,784 || trainable%: 2.0466

Say these points

ΔW = (α/r)·BA with B initialised to zero, so training starts at the pre-trained model.
0.1–2% trainable parameters collapses optimiser memory; adapters are tens of MB.
QLoRA = NF4 4-bit frozen base + double quantisation + paged optimisers; ~30% slower per step.
Targeting all linear layers matters more than increasing rank.
Keep adapters unmerged for multi-tenant hot-swapping; merge for single-model latency.

Avoid these mistakes

Targeting only q_proj and v_proj because a tutorial did.
Assuming QLoRA is faster — it saves memory, not compute.
Merging a bf16 adapter into a 4-bit base without dequantising.

Expect these follow-ups

Why is NF4 better than plain int4 for neural network weights?
How does S-LoRA serve hundreds of adapters concurrently?
What is DoRA and how does it differ from LoRA?

ExpertrlhfdpoalignmentAsked at Anthropic, OpenAI

30-second answer

RLHF has three stages: supervised fine-tuning, training a reward model on human preference pairs, then optimising the policy against that reward with PPO plus a KL penalty to the reference model. DPO proves the optimal RLHF policy has a closed form, which lets you skip the reward model and the RL loop entirely and optimise a simple classification-style loss directly on preference pairs. DPO won on simplicity, stability and cost — it needs two models in memory instead of four.

RLHF, stage by stage

1 — Supervised fine-tuning (SFT)
Train on high-quality demonstration data (prompt → ideal response). This establishes format and basic helpfulness, and gives the reference policy π_ref.
2 — Reward model
Collect pairs (chosen, rejected) from human labellers and train a scalar reward head with the Bradley–Terry loss $- lo g σ (r_{θ} (x, y_{w}) - r_{θ} (x, y_{l}))$ . Only relative preference is learned — the absolute scale is arbitrary.
3 — PPO against the reward model
Maximise $E [r_{θ} (x, y)] - β KL (π_{θ} ∥ π_{re f})$ . The KL term is essential: without it the policy finds adversarial inputs that score highly under the reward model while being nonsense — reward hacking.

The DPO derivation, in one move

The KL-constrained reward maximisation has a known closed-form optimum:

π^{*} (y ∣ x) = \frac{1}{Z ( x )} π_{re f} (y ∣ x) exp (\frac{1}{β} r (x, y))

Rearranging gives the reward implicitly in terms of the policy: $r (x, y) = β lo g \frac{π ( y ∣ x )}{π _{re f} ( y ∣ x )} + β lo g Z (x)$ . Substituting into the Bradley–Terry preference model, the intractable partition function $Z (x)$ cancels because it is identical for the chosen and rejected response. What remains is a plain loss on preference pairs:

L_{D PO} = - E [lo g σ (β lo g \frac{π _{θ} ( y _{w} ∣ x )}{π _{re f} ( y _{w} ∣ x )} - β lo g \frac{π _{θ} ( y _{l} ∣ x )}{π _{re f} ( y _{l} ∣ x )})]

pythonThe entire DPO loss — 12 lines

import torch, torch.nn.functional as F

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps,    ref_rejected_logps, beta=0.1):
    """All inputs: summed log-probs of the full response, shape (batch,)."""
    pi_logratios  = policy_chosen_logps - policy_rejected_logps
    ref_logratios = ref_chosen_logps    - ref_rejected_logps
    logits = beta * (pi_logratios - ref_logratios)

    loss = -F.logsigmoid(logits).mean()

    # Diagnostics worth logging: implicit rewards and preference accuracy
    chosen_r  = beta * (policy_chosen_logps  - ref_chosen_logps).detach()
    rejected_r= beta * (policy_rejected_logps- ref_rejected_logps).detach()
    return loss, {
        "reward_margin":   float((chosen_r - rejected_r).mean()),
        "reward_accuracy": float((chosen_r > rejected_r).float().mean()),
    }

# Compare the machinery:
#   PPO: 4 models in memory, generation loop, ~12 hyperparameters, unstable
#   DPO: 2 models (policy + frozen reference), no generation, 1 key hyperparameter

	RLHF (PPO)	DPO	GRPO
Models in memory	4	2	2 (no critic)
Needs a reward model	Yes	No	Yes (or a verifier)
Online generation	Yes	No	Yes
Key hyperparameters	~12	β	β, group size
Stability	Fragile	Stable	Moderate
Can exceed the preference data	Yes (explores)	No (offline)	Yes
Best for	Frontier alignment with a reward budget	Most applied use cases	Verifiable-reward tasks (maths, code)

Also be ready for GRPO (Group Relative Policy Optimisation), used in DeepSeek's reasoning models: it removes the value network by sampling a group of responses per prompt and using the group's mean reward as the baseline. For tasks with a verifiable reward — maths answers, unit tests passing — this replaces the learned reward model entirely and largely eliminates reward hacking, which is the core idea behind the recent wave of reasoning models.

Say these points

RLHF = SFT → reward model (Bradley–Terry) → PPO with a KL penalty to the reference.
The KL term prevents reward hacking; without it the policy exploits the reward model.
DPO uses the closed-form optimal policy so the partition function cancels — no reward model, no RL.
DPO: 2 models, 1 main hyperparameter, stable. PPO: 4 models, ~12 hyperparameters, fragile.
DPO is offline and cannot exceed its preference data; GRPO uses group baselines with verifiable rewards.

Avoid these mistakes

Describing RLHF as 'training on human-written answers' — it trains on human *rankings*.
Omitting the KL penalty when explaining PPO for RLHF.
Claiming DPO is strictly better; it trades exploration for stability.

Expect these follow-ups

What is reward hacking and how would you detect it in production?
Explain iterative/online DPO and why it helps.
How does constitutional AI reduce the human labelling burden?

Advancedcatastrophic-forgettingfine-tuningdata-mixingAsked at Hugging Face, Cohere

30-second answer

Catastrophic forgetting: gradient descent on a narrow distribution moves weights away from the configuration that encoded general capability, and nothing in the loss penalises that. Fixes ranked by effectiveness: mix 5–30% general instruction data into the fine-tuning set, use LoRA so the base weights are frozen, lower the learning rate and train fewer epochs, and always evaluate on a general benchmark alongside your domain metric.

The mechanism is straightforward. Your loss only measures domain performance. Any weight configuration that minimises it is acceptable to the optimiser, including ones that destroy unrelated capabilities. There is no term in the objective that says 'and stay good at everything else'.

pythonData mixing — the highest-leverage fix

import random

def build_mixed_dataset(domain_data, general_data, general_ratio=0.15, seed=0):
    """15-30% general instruction data is the standard recommendation."""
    n_general = int(len(domain_data) * general_ratio / (1 - general_ratio))
    rng = random.Random(seed)
    mixed = domain_data + rng.sample(general_data, min(n_general, len(general_data)))
    rng.shuffle(mixed)                     # interleave -- never train in blocks
    return mixed

# Measured effect on an 8B model, 3 epochs LoRA r=16:
#
# general_ratio   domain_acc   MMLU    GSM8K   HumanEval
#          0.00        0.912   0.518    0.271      0.183   <- base was 0.651/0.554/0.372
#          0.05        0.908   0.601    0.412      0.288
#          0.15        0.903   0.641    0.523      0.351   <- sweet spot
#          0.30        0.887   0.649    0.548      0.368
#
# 15% general data costs 1 point of domain accuracy and recovers ~25 points of MMLU.

Mitigation	Mechanism	Effectiveness
Mix in general data (15–30%)	Keeps the general distribution in the loss	Highest — do this first
LoRA instead of full FT	Base weights frozen; low-rank update is a bounded perturbation	High, and free
Lower LR + fewer epochs	Smaller total displacement from the base weights	High; 1–3 epochs is usually enough
Early stopping on a general benchmark	Stops before the damage accumulates	High, cheap to add
KL penalty to the base model	Explicitly penalises drift in output distribution	Effective but adds a second forward pass
Model merging / weight averaging	Interpolate fine-tuned and base weights	Surprisingly effective (model soups)
Elastic weight consolidation	Penalise changes to high-Fisher-information weights	Theoretically clean, rarely used at LLM scale

pythonWeight interpolation sweep

import torch, copy

def interpolate(base_sd, ft_sd, lam):
    return {k: lam * ft_sd[k].float() + (1 - lam) * base_sd[k].float()
            for k in base_sd if k in ft_sd and base_sd[k].shape == ft_sd[k].shape}

for lam in (0.0, 0.3, 0.5, 0.7, 0.9, 1.0):
    model.load_state_dict(interpolate(base_sd, ft_sd, lam), strict=False)
    print(f"lambda={lam:.1f}  domain={eval_domain(model):.3f}  mmlu={eval_mmlu(model):.3f}")

# lambda=0.0  domain=0.412  mmlu=0.651     <- pure base
# lambda=0.3  domain=0.788  mmlu=0.644
# lambda=0.5  domain=0.869  mmlu=0.631
# lambda=0.7  domain=0.897  mmlu=0.605     <- good trade-off
# lambda=0.9  domain=0.910  mmlu=0.551
# lambda=1.0  domain=0.912  mmlu=0.518     <- pure fine-tune

Say these points

Forgetting happens because the loss contains no term for preserving general capability.
Mixing 15–30% general instruction data is the single most effective fix.
LoRA bounds the perturbation by freezing the base weights.
Weight interpolation between base and fine-tune recovers capability with zero training.
Always evaluate general benchmarks and safety alongside the domain metric.

Avoid these mistakes

Training many epochs on a narrow dataset at a high learning rate.
Evaluating only the domain metric and declaring success.
Training domain and general data in separate blocks instead of interleaved.

Expect these follow-ups

How does TIES-merging resolve conflicts between multiple task vectors?
How would you fine-tune sequentially on three domains without forgetting?
What is a task vector and how is it computed?

Advanceddata-curationinstruction-tuningsftAsked at Scale AI, Surge

30-second answer

Quality and diversity beat volume decisively. LIMA showed 1,000 carefully curated examples can outperform 50,000 noisy ones for instruction tuning. Target 1,000–10,000 examples for style and format tasks, 10,000–100,000 for genuinely new capability. Good examples are diverse in phrasing and task type, consistent in output format, contain no contradictions, and — critically — no example should require knowledge the model does not have.

Goal	Examples needed	Notes
Output format / schema	500–2,000	Highly consistent formatting matters more than count
Tone and brand voice	1,000–5,000	Diverse topics, consistent register
Domain task adaptation	5,000–50,000	Needs coverage of the real task distribution
New capability	50,000+	Often better solved by a different base model
Preference tuning (DPO)	2,000–20,000 pairs	Pair quality dominates; hard pairs teach most

pythonA dataset quality pipeline

import hashlib, numpy as np
from collections import Counter

def audit_dataset(examples, embed_fn):
    """Run this BEFORE training. Most fine-tuning failures are data failures."""
    report = {}

    # 1. Exact and near duplicates
    hashes = [hashlib.md5(e["output"].encode()).hexdigest() for e in examples]
    report["exact_dupes"] = len(examples) - len(set(hashes))

    E = embed_fn([e["instruction"] for e in examples])
    E = E / np.linalg.norm(E, axis=1, keepdims=True)
    sim = E @ E.T
    np.fill_diagonal(sim, 0)
    report["near_dupes"] = int((sim > 0.95).sum() // 2)

    # 2. Diversity: how spread out are the instructions?
    report["mean_pairwise_sim"] = float(sim[np.triu_indices_from(sim, 1)].mean())

    # 3. Format consistency -- inconsistency here is a top cause of bad fine-tunes
    starts = Counter(e["output"][:20] for e in examples)
    report["most_common_opening"] = starts.most_common(1)[0]

    # 4. Length distribution: outliers dominate the loss and skew the style
    lens = np.array([len(e["output"].split()) for e in examples])
    report["length_p50_p95_max"] = (int(np.percentile(lens, 50)),
                                    int(np.percentile(lens, 95)), int(lens.max()))

    # 5. Refusals accidentally included -- teaches the model to refuse your own domain
    refusal_markers = ("I cannot", "I'm unable", "As an AI", "I don't have access")
    report["accidental_refusals"] = sum(
        any(m in e["output"] for m in refusal_markers) for e in examples)
    return report

# {'exact_dupes': 213, 'near_dupes': 1847, 'mean_pairwise_sim': 0.71,
#  'most_common_opening': ('Sure! Here is the ', 4102),
#  'length_p50_p95_max': (84, 412, 8931), 'accidental_refusals': 96}
#
# Read that as: too repetitive (0.71 mean similarity), 4102 examples share an
# opening (the model will parrot it), one 8931-word outlier will dominate the
# loss, and 96 examples teach it to refuse. Fix ALL of this before training.

Where the data comes from

Mine production logs
The highest-value source: real user requests with outcomes you can grade. Filter for interactions a human confirmed as good. This automatically matches your real distribution — which synthetic data never quite does.
Frontier-model distillation
Generate responses with a strong model, then filter with a rubric-based judge and human spot-checks. Cheap and effective. Check the provider's terms — some prohibit training competing models on their outputs.
Self-instruct with seed diversity
Bootstrap from a few hundred human-written seeds, generate variations, and deduplicate aggressively by embedding similarity. Without hard deduplication you get 50,000 rephrasings of the same 200 tasks.
Human writing for the hard cases
Expensive but irreplaceable for edge cases, refusals-that-should-not-happen, and anything requiring genuine domain judgement. Spend your human budget on the tail, not the head.
Hold out a real evaluation set
Curated by hand, never used for training, representative of production. 100–300 examples is enough and it is the artefact that makes every other decision measurable.

Say these points

LIMA: 1,000 curated examples beat 52,000 noisy ones for instruction tuning.
1k–10k for style/format; 10k–100k for new capability.
Audit for duplicates, diversity, format consistency, length outliers and accidental refusals.
Production logs are the best source because they match the real distribution.
Hold out 100–300 hand-curated eval examples that never touch training.

Avoid these mistakes

Optimising for dataset size instead of diversity and consistency.
Leaving 'As an AI language model' refusals in a domain dataset.
Generating synthetic data without aggressive near-duplicate removal.

Expect these follow-ups

How would you use an LLM judge to filter synthetic training data?
How do you handle multi-turn conversations in an SFT dataset?
What licence and privacy issues arise from training on production logs?

Fine-tuning, LoRA/QLoRA and Preference Optimisation

A stakeholder wants to fine-tune an LLM on their company documentation so it 'knows' their product. What do you tell them?

How to run the conversation

Explain LoRA mathematically. What does QLoRA add, and what rank would you choose?

What QLoRA adds

Explain RLHF and how DPO differs. Why did DPO become popular?

RLHF, stage by stage

The DPO derivation, in one move

After fine-tuning on your domain data, the model gets worse at general tasks. What is happening and how do you fix it?

How would you build a fine-tuning dataset? How much data do you need and what makes an example good?

Where the data comes from

Check your understanding

Hands-on challenge

Run a controlled fine-tuning experiment

Curriculum