LLM Quantization & LoRA Fine-Tuning: running 70B models on consumer hardware

A state-of-the-art 70-billion parameter model (like Meta’s Llama 3 70B) in standard 16-bit floating point (FP16 or BF16) precision requires over 140 Gigabytes of GPU VRAM just to load its weights into memory.

When serving concurrent requests or running long-context generation, KV-cache allocations push total VRAM requirements well beyond 160 GB—demanding multi-GPU clusters composed of dual 80GB Nvidia H100 or A100 GPUs.

To make modern foundation models accessible for local edge deployment, consumer laptops (such as Apple Silicon Mac Studios), and cost-effective cloud inference, AI engineers rely on two essential compression techniques: Quantization and Low-Rank Adaptation (LoRA).

FP16 MODEL SIZE:
70 Billion Parameters × 2 Bytes (16-bit) = 140 GB VRAM (Requires Cluster)

4-BIT QUANTIZED SIZE (AWQ / GGUF):
70 Billion Parameters × 0.5 Bytes (4-bit) = 35 GB RAM (Runs on 1 Mac / GPU)

1. How Quantization Works: From FP16 to INT4

In an uncompressed neural network, every weight is stored as an IEEE 754 16-bit floating-point number:

1 sign bit
5 exponent bits
10 mantissa bits

Quantization maps continuous 16-bit floating points into discrete low-bit integers (e.g. 8-bit INT8 or 4-bit INT4):

FP16 (16-bit)  ➔  Range [-65504 to +65504]  ➔  16 bits/param
INT8 (8-bit)   ➔  Range [-128 to +127]       ➔  8 bits/param  (-50% VRAM)
INT4 (4-bit)   ➔  Range [-8 to +7]           ➔  4 bits/param  (-75% VRAM)

The Outlier Challenge & Activation-Aware Quantization (AWQ)

Naively quantizing all parameters uniformly degrades perplexity because ~1% of weights (salient activation outliers) contain over 90% of the model’s semantic reasoning capability.

Modern quantization algorithms solve this:

AWQ (Activation-Aware Weight Quantization): Observes activation magnitudes during inference and keeps the top 1% critical weights in high precision while compressing the remaining 99% to 4-bit.
GGUF (llama.cpp format): Implements variable k-quant schemes (e.g. Q4_K_M), allocating more bits to attention matrices and fewer bits to feed-forward layers. Supports zero-copy memory mapping (mmap) directly into unified RAM.

2. Low-Rank Adaptation (LoRA & QLoRA)

When adapting a base model to a specific domain (e.g., medical diagnostics, SQL query generation, or internal company documentation), standard Full Fine-Tuning updates all 70 billion parameters.

In full fine-tuning with the AdamW optimizer:

Model weights: 140 GB
Gradients: 140 GB
Optimizer states (momentum + variance): 280 GB
Total VRAM required for training: 560+ GB VRAM

The Math Behind LoRA

LoRA (Low-Rank Adaptation) is based on the insight that the weight update matrix ΔW during fine-tuning has an intrinsically low “rank” (dimension of information change).

Instead of computing a full d × k matrix update W = W0 + ΔW, LoRA freezes the original weights W0 and factors ΔW into two tiny matrices A and B:

ΔW = B × A

Where:
- W0 is frozen: (4096 × 4096) = 16,777,216 parameters
- B is trainable: (4096 × 16)   = 65,536 parameters
- A is trainable: (16 × 4096)   = 65,536 parameters

Total trainable parameters with rank r=16: 131,072 (< 0.8% of base weights!)

     ┌──────────────────────┐
     │  Frozen Base Weights │
     │         W0           │
     │      (4096x4096)     │
     └──────────┬───────────┘
                │
Input X ────────┼──────────────────┐
                │                  │
                ▼                  ▼
             W0 × X            Matrix A (4096x16)
                │                  │
                │                  ▼
                │              Matrix B (16x4096)
                │                  │
                ▼                  ▼
             Output Y  =   W0(X) + (B × A)(X)

QLoRA (Quantized LoRA)

QLoRA takes this one step further by loading the base model W0 in NF4 (NormalFloat 4-bit) and backpropagating gradients through the 4-bit weights into the FP16 LoRA adapter matrices A and B.

This allows developers to fine-tune a 70B parameter model on a single 24GB consumer GPU (like an Nvidia RTX 4090).

3. Practical Comparison & Architecture Cheatsheet

Metric	Full FP16 Fine-Tuning	LoRA (FP16 Base)	QLoRA (4-bit Base)	GGUF / AWQ (Inference Only)
70B Model Memory	560 GB VRAM	160 GB VRAM	24 GB VRAM	38 GB Unified RAM
Hardware Needed	8× 80GB H100	2× 80GB A100	1× RTX 4090 (24GB)	Mac Studio M2/M3 (64GB)
Adapter File Size	140 GB (Full weights)	50 MB - 200 MB	50 MB - 200 MB	N/A
Multi-Tenant Serving	1 model per GPU cluster	Swap adapters dynamically	Swap adapters dynamically	Single static model
Throughput Speed	Baseline	100%	85-90%	150-300% (Batching/vLLM)

4. Key Takeaways

Quantization is for Inference: Use 4-bit AWQ with vLLM or GGUF with Ollama/llama.cpp to run massive models locally or at 1/4th the cloud hosting cost.
LoRA is for Fine-Tuning: Never fully fine-tune foundation models unless pretraining from scratch. LoRA gives identical task specialization while saving 99% of storage and 80% of VRAM.
Dynamic Multi-Tenancy: With LoRA, a single GPU can serve 100 different fine-tuned customer models by routing requests through different tiny 20MB adapter matrices in memory.