LoRA (Low-Rank Adaptation)
At a glance
Definition
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that freezes the pretrained model weights and injects small trainable rank-decomposition matrices into targeted layers of the network. Instead of updating the original weight matrix W directly, LoRA learns two low-rank matrices A and B such that the update ΔW ≈ B·A. The rank r is a small number (typically 4-64) rather than the full model dimension.
The core insight came from the observation that fine-tuning updates tend to have low intrinsic rank — the effective changes to a large weight matrix during fine-tuning can be well-approximated by a much smaller matrix. Rather than training the full 4096×4096 weight matrix, you train a 4096×16 and 16×4096 pair, which multiply out to give an update in the same shape. For a 4096×4096 layer, that's 16.8M parameters vs. 131K — roughly a 128x reduction.
In practice, LoRA is applied to a subset of layers — most commonly the attention query, key, value, and output projection matrices, since these are where fine-tuning tends to have most impact. Modern setups often add LoRA to feed-forward layers too. The rank and alpha hyperparameters give you dials to trade capacity for compute.
LoRA has spawned a family of variants. QLoRA combines LoRA with 4-bit quantization of the base model, enabling fine-tuning of 65B+ models on a single consumer GPU. AdaLoRA, DoRA, and other refinements improve quality or efficiency. In 2026, LoRA is the default choice for most practical fine-tuning — used by every major fine-tuning framework and every open-source community.
Real-world example
How LoRA cuts a 70B fine-tuning job from impossible to overnight
Say you want to fine-tune Llama-4 70B on 5,000 support conversations. Full fine-tuning requires ~280GB of GPU memory just for the model weights, plus another ~600GB for gradient states and optimizer momentum — hundreds of GPUs, days of training. With LoRA (rank=16), you freeze the base model and train only ~50 million adapter parameters (0.07% of total). Memory drops to ~140GB (with careful loading). Training time: overnight on 4 H100s. Result: quality within a few percentage points of full fine-tuning, at a small fraction of the cost.
# LoRA fine-tuning with PEFT and TRL
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
# 1. Load base model (frozen)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-4-8B",
torch_dtype="bfloat16",
)
# 2. Configure LoRA — the key hyperparameters
lora_config = LoraConfig(
r=16, # rank — bigger = more capacity, more cost
lora_alpha=32, # scaling factor (often 2 * r)
target_modules=[ # which layers to adapt
"q_proj", "k_proj",
"v_proj", "o_proj", # all four attention projections
],
lora_dropout=0.05, # regularization
bias="none", # don't train bias terms
task_type="CAUSAL_LM",
)
# 3. Wrap model — original weights frozen, adapters trainable
peft_model = get_peft_model(model, lora_config)
peft_model.print_trainable_parameters()
# Trainable: ~4.2M Total: ~8.0B Ratio: 0.05%
# 4. Train normally with your favorite trainer.
# Save just the LoRA adapters (a few hundred MB, not 16GB).When you'll encounter this
- You're fine-tuning any modern LLM — LoRA is the default choice unless you have a specific reason for full fine-tuning.
- You're running on limited hardware — a single consumer GPU can fine-tune 7-8B models with LoRA, or 65B+ with QLoRA.
- You need to maintain multiple specializations of the same base model — LoRA adapters are small (~100MB) and swappable.
- You want to experiment quickly — LoRA training is 3-10x faster than full fine-tuning, letting you iterate on data and hyperparameters.
- You want to avoid catastrophic forgetting — freezing base weights means the model retains its general capabilities.
How it works
Identify target layers
Pick which layers get LoRA adapters. Most common: attention projections (q_proj, k_proj, v_proj, o_proj). More capacity: add MLP layers (gate_proj, up_proj, down_proj).
Add low-rank matrices
For each target layer with weight matrix W of shape (d_out × d_in), create two matrices: A of shape (r × d_in) initialized randomly, B of shape (d_out × r) initialized to zero. The effective update is B·A.
Freeze base weights
Set requires_grad=False on all original model parameters. Only A and B matrices are trainable.
Compute forward pass
During inference, the effective weight is W + (α/r)·B·A. The scaling factor α/r controls how much the adapter contributes. At training start, B is zero, so the model behaves identically to the base.
Train adapters
Backpropagation only updates A and B. Gradient memory and optimizer state (for Adam: 8 bytes per param × 2 states) are dramatically smaller.
Save adapters
Save only the LoRA adapter weights, not the base model. Adapter files are typically 50-500MB depending on rank and target layers.
Deploy
At inference, load the base model once, then load the adapter on top. Or merge the adapter into the base weights (W + BA becomes the new W) for slightly faster inference at the cost of adapter modularity.
Common misconceptions
Usually within 1-3% on standard benchmarks. For most practical tasks the difference is negligible. Cases where full fine-tuning wins: extremely low-resource languages, very long training runs, or when you specifically need to change knowledge the base model has wrong.
No. Higher rank means more parameters, more memory, and more compute — but returns diminish quickly. Rank 8-32 is a sweet spot for most tasks. Sometimes rank 4 works. Only very complex tasks benefit from rank 128+.
QLoRA = LoRA + 4-bit quantization of the base model. QLoRA lets you fit much larger models in memory but has slightly different quality trade-offs. Regular LoRA keeps the base model in fp16 or bf16.
Related terms
Where you'll see this in practice
Learn more
- Read the deep-dive: How LoRA Works
- Compare LoRA vs QLoRA
- Take the Master Fine-Tuning path
- Browse Fine-tuning Methods category
Sources & further reading
- LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021 (original LoRA paper, Microsoft). Read →
- QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023. Read →
- DoRA: Weight-Decomposed Low-Rank Adaptation — Liu et al., 2024. Read →
- PEFT Documentation — HuggingFace PEFT library. Read →
Frequently asked about LoRA
Rank 8-16 for most tasks. Higher rank (32-64) for complex tasks with lots of training data. Rank 4 sometimes works for narrow adaptation. Test on a small validation set — the optimal rank varies by task and dataset size more than by model size.
Start with all four attention projections (q, k, v, o). If you need more capacity, add MLP layers (gate_proj, up_proj, down_proj). Adding more targets improves quality but increases parameter count and memory.
Alpha is a scaling factor — the effective update is (α/r)·BA. A common convention is α = 2r (e.g., rank=16, alpha=32). Higher alpha means the adapter has more influence. If you're seeing under-fitting, try increasing alpha; over-fitting, decrease.
Yes. Because W_new = W + BA and BA is small-rank, you can compute W_new once and use it as a normal weight matrix. This eliminates adapter overhead at inference. Downside: you lose the ability to swap adapters and you get a full-size checkpoint again.
Yes. LoRA has been applied to Stable Diffusion (image generation), Whisper (speech), and vision transformers. Anywhere you have a large pretrained transformer, LoRA is a viable adaptation method. The Civitai community, for example, distributes thousands of LoRA adapters for image models.