How to Choose a GPU for Training and Deploying LLMs

The One-Line Principle

Before picking a GPU, remember: training and inference need different things.

• Training wants three things: compute (FLOPS) + VRAM + inter-GPU interconnect. Compute decides how fast it trains, VRAM decides how big a model it can train, and interconnect decides multi-GPU efficiency.
• Inference wants two things: VRAM capacity + memory bandwidth. Capacity decides how big a model fits, and bandwidth decides token generation speed—inference speed is often bottlenecked by bandwidth.

So there is no universal answer to “which GPU is best”—it depends on where you use it.

First, Separate the Training Phase from the Inference Phase

Building “your own model” means you will have both training and inference workloads, but their frequency and cost structure are completely different:

Stage	Category	Frequency	Typical load	Cost nature
Data prep (cleaning, rebuilding samples)	Data engineering	Periodic	CPU, memory	Cheap
SFT / DPO fine-tuning	Training	Occasional (days per run)	GPU compute-bound	Rent, stop when done
User chat Q&A	Inference	7×24 continuous	VRAM bandwidth-bound	Continuous, fixed cost

Two key realizations:

• Fine-tuning is training too (LoRA/QLoRA is just parameter-efficient training). If you want to “train your own model,” you cannot avoid training.
• Inference is the long-term cost center; training is a one-off investment per stage.

This directly drives different GPU strategies: training is rented on demand and released when done; inference must stay online long-term and be cost-optimized.

1. VRAM Is the Hard Gate

Everything starts with VRAM. Estimate how much the weights take with this formula:

VRAM ≈ parameter count × bytes per parameter

Bytes per parameter by precision:

Precision	Bytes per param	27B model weights
FP16 / BF16	2	~54 GB
FP8 / INT8	1	~27 GB
INT4	0.5	~14 GB

Beyond weights, add KV cache (grows with conversation length) and activations. In practice, reserve 20%–40% extra VRAM for inference.

Conclusion: 27B BF16 needs a single card with ≥80GB; quantized to 8-bit, a 48GB card can run it.

2. VRAM-to-Capability Table

VRAM	Typical cards	What you can do
24GB	RTX 4090	QLoRA fine-tune 7B–14B; single-user inference for small models
32GB	RTX 5090	QLoRA up to 14B; roomier small-model inference
48GB	L40S / RTX 6000 Ada / A6000	27B quantized inference; LoRA fine-tune 7B–14B
80GB	A100 / H100	27B BF16 inference; full fine-tune 7B–14B
96GB	H20 (China-specific)	Good for large-model inference, but weak compute—poor value for training
141GB	H200	70B-class inference; high-throughput batching
180GB	B200	Top-tier training/inference, and the most expensive

If your base is an MoE (Mixture-of-Experts) architecture—say 26B total with only 3.8B active—VRAM and throughput are far friendlier, making it the value pick for scaling a chat product.

3. Picks by Scenario

1) Experiments / getting started: prove the pipeline

The goal is to run the whole “base + data + SFT” loop:

• 1× RTX 4090 24G (or 5090 32G) is enough for QLoRA plus small-model inference;
• But at this stage renting is the cheapest option—do not rush to buy.

2) Small teams, train + serve on one box

• 2× L40S 48G or 1× A100 80G;
• Quantized 27B inference plus LoRA fine-tuning, one card covering both.

3) Production / scale

• 4–8× H100 / H200 (with NVLink): high-QPS inference plus full fine-tuning;
• To run several-hundred-billion MoE models, you need an 8-GPU H20/H200-class cluster.

Interconnect is the hidden killer: with multi-GPU tensor parallelism (TP), no NVLink means a serious slowdown. Prefer NVLink-equipped systems or modules of the same model.

4. Rent or Buy: Rent First

This is the most overlooked and most cost-impacting rule.

• Buying GPUs is a fixed cost—idle capacity is pure waste;
• For selection, experiments, and temporary training, hourly cloud billing is cheapest;
• Only when GPU utilization is consistently high (a rule of thumb: >60%–70%) does buying pay off.

Rental options:

• China: AutoDL (cheap, lots of 4090/A100), Alibaba Cloud, Volcano Engine, Tencent Cloud
• Overseas: RunPod, Lambda, Vast.ai

Advice: rent through the entire selection and SFT phase; consider building your own only once the business is stable and utilization is predictable.

5. China-Market Caveats

• H20: large VRAM (96G) suits inference, but compute is cut down—poor value for training. Do not make it your fine-tuning workhorse;
• RTX 4090 / 5090: consumer cards with great value, good for individuals/small teams starting out, but not for multi-GPU large-scale training;
• A800 / H800 and similar: depends on supply channels and price;
• Domestic accelerators: the ecosystem is still catching up—before choosing, confirm support in training/inference frameworks (vLLM, LLaMA-Factory, etc.).

6. The Default Answer

• Just proving the pipeline → rent 1× A100 80G (or 2× L40S 48G): quantized 27B inference + LoRA fine-tuning in one go;
• Going to production → start with 4× H100/H200, then scale by concurrency and QPS;
• Budget-constrained individual → RTX 4090/5090 with cloud GPUs as a fallback.

Summary

• Training cares about compute + VRAM + interconnect; inference cares about VRAM capacity + bandwidth;
• 27B BF16 needs a single card with ≥80GB; quantized, 48GB works;
• Rent first, buy later—buy only at high utilization;
• Multi-GPU training must account for NVLink;
• MoE and quantization are the two key cost levers.

In one line: do not ask “which GPU is strongest”—first ask “which phase am I in, how big a model, and how high is utilization.”