LoRA and QLoRA fine-tuning: when it is worth it and what it costs
How LoRA and QLoRA work, with trainable parameters, GPU memory and training cost worked out for Mistral-7B, plus evaluation, serving and failure modes.
LoRA made fine-tuning a 7B model something you can do on one rented GPU in an afternoon. That is also the problem. It is now cheap enough that teams fine-tune before they have checked whether a better prompt would have done the same job, and then they ship an adapter that is better at the output format and worse at everything else.
This post covers how LoRA and QLoRA work, the parameter count and GPU memory worked out line by line for a real model config, what a training run costs, how to evaluate it so the result means something, how to serve it, and the ways it goes wrong. At the end there is a small Python script (standard library only) that does all the arithmetic for any Hugging Face config.
Most LoRA tutorials give a LoraConfig block and a single memory figure, and the figures disagree because they depend on sequence length, batch size and checkpointing. Here the arithmetic is shown, so you can redo it for your own setup.
If you have not yet decided whether fine-tuning is the right tool at all, read Fine-tuning, RAG or prompting? A decision guide first. This post assumes you have a reason to fine-tune and want to do it without wasting money.
How LoRA works#
A transformer layer has weight matrices such as the attention query projection W_q. Full fine-tuning updates every number in every one of them. LoRA (Hu et al., 2021) freezes the pretrained weight W0 and learns the change as the product of two thin matrices:
frozen trainable
h = W0 · x + (alpha / r) · B · A · x
W0 : d_out x d_in (e.g. 4096 x 4096 = 16.8M numbers)
A : r x d_in (16 x 4096 = 65,536)
B : d_out x r (4096 x 16 = 65,536)
x (d_in) ──► W0 ─────────────────────────────┐
│ (+) ──► h (d_out)
└──► A ──► r-dim ──► B ──► x alpha/r┘
The paper's argument is that the weight change needed to adapt a pretrained model has a low "intrinsic rank", so a rank of 8 or 16 can capture most of it. Three details matter in practice.
Initialisation. A starts random and B starts at zero, so B·A is zero and the model behaves exactly like the base model at step 0. PEFT does the same by default; its LoraConfig reference says the adapter "will be a no-op" before training.
Scaling. The update is multiplied by alpha / r. The LoRA paper notes that with Adam, tuning alpha is roughly the same as tuning the learning rate, so the authors set alpha once and left it. The practical effect: if you double r and keep alpha fixed, each unit of update gets half the weight, and your old learning rate may now be too low. PEFT also offers use_rslora=True, which scales by alpha / sqrt(r) instead, for when you want to sweep rank without retuning the learning rate.
Which modules. The original paper adapted only W_q and W_v in most experiments, and on GPT-3 found that pair worked best for a fixed parameter budget. The QLoRA paper (Dettmers et al., 2023) later found that LoRA on all linear layers of the transformer block was needed to match full 16-bit fine-tuning. In PEFT that is target_modules="all-linear", which picks every linear layer except the output head. I would default to all-linear; the extra parameters cost little memory, as the numbers below show.
After training you can add B·A into W0 and get a normal model back. That is why the LoRA paper says there is no extra inference latency.
Trainable parameters for a real model#
I use Mistral-7B-v0.1 because its config is public without a login. The relevant fields:
hidden_size 4096
intermediate_size 14336
num_hidden_layers 32
num_attention_heads 32
num_key_value_heads 8 (grouped-query attention)
vocab_size 32000
tie_word_embeddings false
With 32 heads of size 128 and 8 key/value heads, the K and V projections output 8 x 128 = 1024 values, not 4096. That changes their LoRA size. Per decoder layer:
| Module | Shape (in x out) | Base params | LoRA params at r=16: r x (in + out) |
|---|---|---|---|
| q_proj | 4096 x 4096 | 16,777,216 | 131,072 |
| k_proj | 4096 x 1024 | 4,194,304 | 81,920 |
| v_proj | 4096 x 1024 | 4,194,304 | 81,920 |
| o_proj | 4096 x 4096 | 16,777,216 | 131,072 |
| gate_proj | 4096 x 14336 | 58,720,256 | 294,912 |
| up_proj | 4096 x 14336 | 58,720,256 | 294,912 |
| down_proj | 14336 x 4096 | 58,720,256 | 294,912 |
| Per layer | 218,103,808 | 1,310,720 |
Thirty-two layers give 6.98B parameters in the block linears. Add the input embedding (32000 x 4096 = 131M), the untied output head (another 131M) and the RMSNorm weights, and the total comes to 7,241,732,096. LoRA at r=16 on all linear layers is 32 x 1,310,720 = 41,943,040 trainable parameters, 0.58% of the model.
The script below prints three common settings:
| Setting | Trainable params | % of model | Adapter file (bf16) |
|---|---|---|---|
| q,v only, r=8 | 3,407,872 | 0.047% | ~7 MB |
| all linear, r=16 | 41,943,040 | 0.58% | ~84 MB |
| all linear, r=64 (QLoRA paper's setting) | 167,772,160 | 2.3% | ~336 MB |
The adapter file is tiny next to the 14.5 GB base checkpoint. That size difference is what makes multi-adapter serving possible later.
GPU memory: full fine-tune vs LoRA vs QLoRA#
Training memory has four parts: weights, gradients, optimizer states and activations. The first three come from the ZeRO paper's accounting for mixed-precision Adam: 2 bytes for the bf16 weight, 2 bytes for its gradient, and 12 bytes for the fp32 master copy plus Adam's two moment estimates. That is 16 bytes per trainable parameter.
The difference between the three methods is which parameters pay the 16 bytes.
Full fine-tune. All 7.24B parameters are trainable: 7.24B x 16 bytes = 116 GB before activations. That does not fit on one 80 GB card. You need several GPUs with sharded optimizer states (ZeRO or FSDP), or an 8-bit optimizer plus offloading.
LoRA on a bf16 base. The base weights are frozen, so they cost only 2 bytes each (14.5 GB) and have no gradients or optimizer state. The 42M LoRA parameters pay the full 16 bytes: about 0.67 GB.
QLoRA. The frozen base is stored in 4-bit NormalFloat (NF4). The paper quantizes in blocks of 64 weights, each block with its own scale. Stored as fp32, those scales add 32/64 = 0.5 bits per parameter. Double quantization stores the scales in 8 bits with a second level of constants, which brings the overhead down to 0.127 bits per parameter. So a quantized weight costs about 4.127 bits. I assume the embeddings, output head and norms stay in bf16, which matches common bitsandbytes setups. The block linears come to 6.98B x 4.127 / 8 = 3.6 GB, plus 0.52 GB for embeddings and head.
Activations are the part most guides skip. Korthikanti et al. estimate activation memory per transformer layer at s·b·h·(34 + 5·a·s/h) bytes for a standard GPT block. With flash attention the second term mostly disappears, leaving about 34·s·b·h. For seq 2048, batch 1, hidden 4096 that is 285 MB per layer, or 9.1 GB for 32 layers. Gradient checkpointing keeps only each layer's input (2 bytes x s x h per layer, 537 MB total) and recomputes one layer at a time in the backward pass. The fp32 logits and their gradient add 2 x 2048 x 32000 x 4 bytes = 0.52 GB. Mistral's MLP differs from a GPT block, so read the 34 as a rough constant, not an exact one.
Put together, for r=16 all-linear, seq 2048, batch 1, checkpointing on:
| Setup | Weights | Grads | Optimizer | Activations | Total (GB) |
|---|---|---|---|---|---|
| Full fine-tune (bf16 + Adam) | 14.5 | 14.5 | 86.9 | 1.35 | 117.2 |
| LoRA (bf16 base) | 14.6 | 0.08 | 0.50 | 1.35 | 16.5 |
| QLoRA (NF4 base) | 4.2 | 0.08 | 0.50 | 1.35 | 6.1 |
These are floors, not budgets. The CUDA context, the allocator's fragmentation and temporary buffers (QLoRA dequantizes each weight block to bf16 for the matmul) all come on top. The QLoRA paper measured the 4-bit LLaMA-7B base at 5,048 MB and the whole 7B setup at 6.9 GB for batch 1 and sequence length 512, which is higher than my formula would give for that model. So I plan for a 24 GB card for QLoRA at 2k context, and 40-48 GB for bf16 LoRA. If you turn off checkpointing, add roughly 8 GB of activations at seq 2048, and that number grows linearly with sequence length and batch size.
The QLoRA paper makes the same point about where memory goes. For LLaMA-7B with LoRA weights at 0.2% of the model, the LoRA parameters took 26 MB while the LoRA input gradients took 567 MB at batch size 1. Cutting rank from 16 to 8 saves almost nothing. Shortening sequences or enabling checkpointing saves a lot. The paper's third trick, paged optimizers, moves optimizer state to CPU memory during memory spikes; the authors needed it for 33B and 65B models, and on a 7B model it is a safety net.
Dataset size and format#
The QLoRA paper has the most useful single data point here. On their chatbot benchmark, a 9k-example dataset (OASST1) beat a 450k-example dataset (FLAN v2). Quality and fit to the task mattered more than size. For narrow tasks (classify a ticket, extract fields into a schema, write in a house style), my rule of thumb is to start with 500 to 2,000 clean examples, see whether the eval moves, then add more where the eval shows errors. That is my starting point, not a measured threshold.
Use the chat format the model will see at inference, one example per JSONL line:
{"messages": [
{"role": "system", "content": "Extract the claim fields as JSON."},
{"role": "user", "content": "Policy 44-1902, water damage on 3 Oct, est. Rs 1.2 lakh"},
{"role": "assistant", "content": "{\"policy\": \"44-1902\", \"peril\": \"water\", \"date\": \"2026-10-03\", \"amount_inr\": 120000}"}
]}
Three rules save the most time:
- Apply the model's own chat template when tokenising, and use the same template at serving time. A mismatch here looks like a model that "forgot" its training.
- Compute loss only on assistant tokens. Training on the system and user text spends capacity on predicting your prompts.
- Split train, validation and test before you clean or deduplicate, and deduplicate across the splits. A near-duplicate of a test example in the training set inflates every score that follows.
What a training run costs#
Compute is the easy part to estimate. A rough rule is 6 FLOPs per model parameter per training token: 2 for the forward pass, 2 for gradients with respect to activations, and 2 for the recomputed forward under checkpointing. LoRA skips the gradients for the frozen weights but still backpropagates through them, so it saves less per token than people expect: full fine-tuning with checkpointing is about 8 FLOPs per parameter per token because it also computes weight gradients, so LoRA saves roughly a quarter. The big saving is that you need one GPU instead of several.
Worked example: 5,000 examples x 600 tokens x 3 epochs = 9M tokens. 6 x 7.24B x 9M = 3.9 x 10^17 FLOPs. An NVIDIA A100 is rated at 312 TFLOPS dense bf16. Assuming 35% utilisation (my assumption; small batches and LoRA's extra kernels keep it low), that is about 1 GPU-hour. QLoRA adds dequantization work, so expect it to run slower than bf16 LoRA on the same card.
Multiply by your provider's hourly rate. The GPU bill for a 7B LoRA run is usually small next to two other costs: building and checking the dataset, and the hyperparameter sweep. The QLoRA authors ran three seeds per configuration in their LoRA rank study. If you sweep 3 learning rates x 2 ranks x 2 seeds, that is 12 runs, so budget for 12 GPU-hours, not 1.
Evaluation: three arms and a regression set#
A fine-tune that beats the base model with a one-line prompt has proved almost nothing. The comparison that matters is against the best you can do without training. I run three arms on the same held-out test set:
held-out test set (never trained on, never tuned on)
│
┌─────────────────────────────┼──────────────────────────────┐
▼ ▼ ▼
A. base model, B. base model, best prompt C. base + LoRA adapter,
minimal prompt (instructions, 5-10 shots, same minimal prompt
JSON mode / schema)
│ │ │
└──────────────► same scorer, same metrics ◄─────────────────┘
task metric · format validity · cost/latency per call
plus D. general regression set (100-300 unrelated prompts): base vs adapter
What I look for:
- C vs B is the real result. If the adapter beats the best prompt by a margin you would bet on, fine-tuning paid for itself. If B and C are within noise, ship B and skip the training pipeline.
- Split the metric. Report format validity (does the JSON parse, does it match the schema) separately from content correctness (are the field values right). See the format-only gains failure below.
- Check variance. With a 200-example test set, one example is 0.5 points. Run the eval on at least two training seeds before you claim a 2-point gain.
- Regression set D. A few hundred prompts from outside your task: general questions, a different language, a refusal case. This is how you catch forgetting.
For building the scorer and the test set, Evaluating an AI agent in Python has a scorer and test runner you can reuse, and free resources to learn LLM evals lists where to go deeper.
Failure modes#
Catastrophic forgetting. The adapter learns your task and loses general skill. Biderman et al. (2024), "LoRA Learns Less and Forgets Less", compared LoRA and full fine-tuning on code and maths. In standard low-rank settings LoRA substantially underperformed full fine-tuning on the target domain, but it kept more of the base model's performance outside it. So LoRA reduces forgetting without removing it. Regression set D is how you measure it. If it drops, try a lower learning rate, fewer epochs, or mix 5-10% general instruction data into training.
Overfitting small data. With 500 examples and 3 epochs, the model sees each example three times. The symptom is training loss heading towards zero while validation loss turns up after the first epoch, and outputs that copy phrases from training examples. Stop on validation loss, use one or two epochs, and add LoRA dropout (the QLoRA paper used 0.1 for models up to 13B).
Format-only gains. This false win is easy to miss. The adapter scores 20 points higher than the base model, and all of it is format: the base model wrapped JSON in prose, the adapter does not. Content accuracy is unchanged. Structured output modes, a schema in the prompt and a few examples often close the same gap with no training. This is why arm B includes JSON mode and why format validity is reported separately.
Template and tokeniser drift. Training with one chat template and serving with another, or a missing end-of-sequence token so the model never learns to stop. The symptom is rambling output after the right answer.
Quantization mismatch. You trained QLoRA on an NF4 base, then merge into a bf16 base for serving. The adapter learned to correct a slightly different set of weights. PEFT's docs warn that on quantized models, merging re-quantizes the base weight, so unmerging is not exact. Re-run your eval on the exact artifact you deploy, not on the training checkpoint.
Serving: merge or keep adapters separate#
Option 1: merge Option 2: multi-adapter
───────────────── ─────────────────────────
W = W0 + (alpha/r)·B·A one base model in GPU memory
one 14.5 GB model per task + N adapters (7-336 MB each)
zero extra latency request picks adapter by name
any inference server works small per-token overhead
N tasks = N full copies N tasks = 1 base + N small files
Merge with PEFT's merge_and_unload() when you have one task, or a few with enough traffic to justify their own deployment. The PEFT LoRA reference notes it returns a new model rather than modifying in place, so assign the result.
Keep adapters separate when you have many tasks or many customers on one base model. The LoRA paper gives the storage case for GPT-3: 100 adapted models take about 354 GB as adapters on one base, against about 35 TB as full copies. vLLM serves this with --enable-lora and --lora-modules name=path. Each adapter shows up as a model name in the OpenAI-compatible API, so a client picks one with the model field. Set --max-lora-rank to your largest adapter's rank, since a higher value wastes memory. The LoRA paper points out the cost of merging: once B·A is folded into W, you cannot batch requests for different tasks in one forward pass. Unmerged adapters avoid that at a small compute cost per token.
If you also care about tokens per second and cost per request after deployment, the LLM cost and latency tracker post shows how I measure both.
When LoRA is worth it#
| Situation | LoRA / QLoRA? | Why |
|---|---|---|
| Model lacks facts that change weekly | No, use RAG | Training does not update reliably and is stale on the next change |
| Output format is wrong, content is right | Usually no | Try structured output and few-shot first; check for format-only gains |
| Consistent style, tone or label set across thousands of calls | Yes | Hard to hold with prompts alone; easy to measure |
| Long prompt (many shots) repeated on every call at high volume | Often yes | An adapter can replace the shots and cut input tokens per call |
| Small model must match a larger model on one narrow task | Yes | Distil the large model's outputs into a 7B adapter |
| Fewer than ~200 good examples and no eval set | Not yet | Build the eval set first; you cannot tell if training helped |
| Need a new language or large new domain | LoRA may be too small | Biderman et al. found LoRA well behind full fine-tuning on code and maths |
The longer version of this decision, including RAG and the order to try things in, is in the fine-tuning vs RAG vs prompting guide.
Runnable: the parameter and memory calculator#
This is the script that produced every table above. It uses the standard library only; it does not train or load a model. Pass a path to any Llama- or Mistral-style config.json to run it on another model.
"""LoRA / QLoRA parameter and memory estimator. Standard library only.
Every number is an estimate. Measure peak memory on your own GPU before buying hardware.
"""
import json
import sys
GB = 1e9
# Mistral-7B-v0.1, copied from its config.json on Hugging Face
MISTRAL_7B = {
"hidden_size": 4096, "intermediate_size": 14336, "num_hidden_layers": 32,
"num_attention_heads": 32, "num_key_value_heads": 8, "vocab_size": 32000,
"tie_word_embeddings": False,
}
def linear_shapes(cfg):
"""(in_features, out_features) for every linear layer in one decoder block."""
h, ff = cfg["hidden_size"], cfg["intermediate_size"]
kv = h // cfg["num_attention_heads"] * cfg["num_key_value_heads"]
return {
"q_proj": (h, h), "k_proj": (h, kv), "v_proj": (h, kv), "o_proj": (h, h),
"gate_proj": (h, ff), "up_proj": (h, ff), "down_proj": (ff, h),
}
def base_params(cfg):
h, layers, vocab = cfg["hidden_size"], cfg["num_hidden_layers"], cfg["vocab_size"]
block_linear = sum(i * o for i, o in linear_shapes(cfg).values()) * layers
norms = (2 * layers + 1) * h
embed = vocab * h
head = 0 if cfg["tie_word_embeddings"] else vocab * h
return {"block_linear": block_linear, "embed_head": embed + head, "norms": norms,
"total": block_linear + embed + head + norms}
def lora_params(cfg, r, targets):
"""Each adapted W (d_in x d_out) gets A (r x d_in) and B (d_out x r): r * (d_in + d_out)."""
shapes = linear_shapes(cfg)
per_layer = sum(r * (shapes[t][0] + shapes[t][1]) for t in targets)
return per_layer * cfg["num_hidden_layers"]
def activations_bytes(cfg, seq, batch, checkpointing):
"""Rough. Korthikanti et al. give ~34*s*b*h bytes per layer for a GPT block with
flash attention (the 5as/h attention-score term drops out). With full checkpointing
we keep each layer's 2-byte input plus one layer being recomputed. The fp32 logits
and their gradient (2 * s * b * vocab * 4 bytes) come on top."""
s, b, h, L = seq, batch, cfg["hidden_size"], cfg["num_hidden_layers"]
logits = 2 * s * b * cfg["vocab_size"] * 4
if checkpointing:
return 2 * s * b * h * L + 34 * s * b * h + logits
return 34 * s * b * h * L + logits
def memory_table(cfg, r, targets, seq, batch):
p = base_params(cfg)
n, n_lora = p["total"], lora_params(cfg, r, targets)
# QLoRA: NF4 = 4 bits, + 0.127 bits/param of quantization constants with double
# quantization (QLoRA paper, section 3). Assumed: embeddings, lm_head, norms stay bf16.
nf4_bytes = p["block_linear"] * (4 + 0.127) / 8 + (p["embed_head"] + p["norms"]) * 2
# Trainable params in mixed precision with Adam: 2 (bf16 weight) + 2 (bf16 grad)
# + 12 (fp32 master weight, Adam m, Adam v) = 16 bytes/param (ZeRO paper).
rows = {
"full fine-tune (bf16 + Adam)": {
"weights": n * 2, "grads": n * 2, "optimizer": n * 12,
"activations": activations_bytes(cfg, seq, batch, checkpointing=True)},
"LoRA (bf16 base)": {
"weights": n * 2 + n_lora * 2, "grads": n_lora * 2, "optimizer": n_lora * 12,
"activations": activations_bytes(cfg, seq, batch, checkpointing=True)},
"QLoRA (NF4 base)": {
"weights": nf4_bytes + n_lora * 2, "grads": n_lora * 2, "optimizer": n_lora * 12,
"activations": activations_bytes(cfg, seq, batch, checkpointing=True)},
}
return p, n_lora, rows
def train_hours(n_params, tokens, peak_tflops, mfu, flops_per_param_token=6):
"""~6 FLOPs per param per token: 2 forward, 2 for activation gradients, 2 for the
recomputed forward under checkpointing. LoRA skips frozen weight gradients but pays
for recomputation, so 6 is a fair rough number for both."""
flops = flops_per_param_token * n_params * tokens
return flops / (peak_tflops * 1e12 * mfu) / 3600
def main():
cfg = json.load(open(sys.argv[1])) if len(sys.argv) > 1 else MISTRAL_7B
r, seq, batch = 16, 2048, 1
all_linear = list(linear_shapes(cfg))
p, n_lora, rows = memory_table(cfg, r, all_linear, seq, batch)
print(f"Base parameters: {p['total']:,} ({p['total'] / 1e9:.2f}B)")
for label, targets, rank in [("q,v only, r=8", ["q_proj", "v_proj"], 8),
("all linear, r=16", all_linear, 16),
("all linear, r=64", all_linear, 64)]:
k = lora_params(cfg, rank, targets)
print(f" LoRA {label:<18} {k:>12,} trainable ({100 * k / p['total']:.3f}%)"
f" adapter file ~{k * 2 / 1e6:.0f} MB in bf16")
print(f"\nTraining memory estimate, r={r} all-linear, seq={seq}, batch={batch}, checkpointing on")
print(f"{'setup':<30}{'weights':>9}{'grads':>8}{'optim':>8}{'activ':>8}{'total':>8} (GB)")
for name, m in rows.items():
total = sum(m.values())
print(f"{name:<30}{m['weights'] / GB:>9.1f}{m['grads'] / GB:>8.2f}"
f"{m['optimizer'] / GB:>8.2f}{m['activations'] / GB:>8.2f}{total / GB:>8.1f}")
off = activations_bytes(cfg, seq, batch, checkpointing=False) / GB
print(f"\nActivations without checkpointing at seq={seq}: ~{off:.1f} GB")
tokens = 5000 * 600 * 3 # examples * avg tokens * epochs
hrs = train_hours(p["total"], tokens, peak_tflops=312, mfu=0.35)
print(f"Compute for {tokens / 1e6:.0f}M training tokens on one A100 (312 TFLOPS bf16 peak,"
f" 35% utilisation assumed): ~{hrs:.1f} GPU-hours")
if __name__ == "__main__":
main()
Output:
Base parameters: 7,241,732,096 (7.24B)
LoRA q,v only, r=8 3,407,872 trainable (0.047%) adapter file ~7 MB in bf16
LoRA all linear, r=16 41,943,040 trainable (0.579%) adapter file ~84 MB in bf16
LoRA all linear, r=64 167,772,160 trainable (2.317%) adapter file ~336 MB in bf16
Training memory estimate, r=16 all-linear, seq=2048, batch=1, checkpointing on
setup weights grads optim activ total (GB)
full fine-tune (bf16 + Adam) 14.5 14.48 86.90 1.35 117.2
LoRA (bf16 base) 14.6 0.08 0.50 1.35 16.5
QLoRA (NF4 base) 4.2 0.08 0.50 1.35 6.1
Activations without checkpointing at seq=2048: ~9.7 GB
Compute for 9M training tokens on one A100 (312 TFLOPS bf16 peak, 35% utilisation assumed): ~1.0 GPU-hours
Two limits of the script. It assumes a Llama/Mistral layout (separate q, k, v, o and a gated MLP), so a model with fused QKV or a mixture-of-experts MLP needs its own linear_shapes. And the activation term is the roughest number in it. Run one training step with torch.cuda.max_memory_allocated() on your real data before you size hardware.
My order of work for a LoRA project: build the held-out test set and arm B (best prompt) first. Fine-tune only if B falls short on the metric you care about. Train QLoRA r=16 all-linear for one or two epochs, compare C against B and D against the base model, and only then decide between merging and multi-adapter serving.
Sources#
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021), for the B·A update, initialisation, alpha/r scaling, W_q/W_v results, no added latency, and the GPT-3 storage numbers.
- QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023), for NF4, double quantization (0.5 to 0.127 bits per parameter), paged optimizers, the all-linear finding, r=64 and alpha=16 settings, the 567 MB vs 26 MB memory breakdown, and OASST1 vs FLAN v2.
- PEFT LoraConfig reference and PEFT LoRA developer guide, for
target_modules="all-linear",use_rslora, default initialisation andmerge_and_unload(). - Transformers bitsandbytes docs, for NF4, nested quantization and bf16 compute dtype.
- Mistral-7B-v0.1 config.json, for the model dimensions.
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, for 16 bytes per parameter with mixed-precision Adam.
- Reducing Activation Recomputation in Large Transformer Models (Korthikanti et al.), for the per-layer activation memory formula.
- LoRA Learns Less and Forgets Less (Biderman et al., 2024), for LoRA vs full fine-tuning on target-domain learning and forgetting.
- vLLM LoRA adapters, for multi-adapter serving flags.
- NVIDIA A100, for the 312 TFLOPS bf16 figure.