How to Choose a Base Model for Chat: A Practical Selection Checklist

Why the Base Model Matters So Much in Chat

When building a chat—especially a companion—product, the first instinct is “which model is the strongest?” But what really decides whether you can keep going long-term is whether you picked the right base model.

Three reasons:

You will fine-tune on top of it. The base sets the ceiling and the “moldability”—some models slide into a persona easily, others stay stuck in customer-service tone no matter what.
It drives your inference cost. Chat is a high-frequency, long-conversation workload with heavy token usage, so base size directly determines the bill.
It decides whether you can own your stack. To avoid being deprecated or repriced by a vendor, you need a model with open weights.

So this post is not about “which model is strongest”—it is about how to choose a base that fits chat, can be fine-tuned, and stays within budget.

1. Six Selection Criteria

1. License: can you commercialize and self-host it

This is the hard gate. Closed models (Claude, GPT, Gemini, etc.) cannot be self-hosted—rule them out. Among open-weight models, prefer permissive licenses like Apache-2.0 / MIT, and note that some “open” models carry extra commercial restrictions. Open the model card’s license and check it line by line.

2. English plus roleplay/creative writing ability

What chat cares about is conversation quality, tone, and character, not math or code. Do not look only at overall scores:

Check LMArena’s Creative Writing and Multi-Turn categories;
Reference roleplay benchmarks (EQ-Bench, RoleBench, and similar);
A model that tops general leaderboards may not be good at chatting.

3. Size vs. inference cost

Chat is a heavy workload, and size equals cost:

Dense models: 12B–31B is the sweet spot between quality and cost;
MoE models: large total parameters but small active parameters (e.g., 25B total, only 3.8B active), giving inference speed close to a small model—the value pick for scaling a companion.

4. An Instruct version plus a healthy fine-tuning ecosystem

You are going to do SFT, so:

There must be an Instruct / Chat version ready for a second round of fine-tuning;
Look at how many community finetunes and adapters exist—a large number means mature tooling and more troubleshooting resources;
Confirm support in LLaMA-Factory / Unsloth / vLLM / Axolotl (verify MoE LoRA support especially).

5. Native system prompt support

The persona is the soul of a chat product. A model with native system role support controls persona more cleanly and breaks character less.

6. Multi-turn stability and long context

Companion products fear two things above all: persona drift and forgetting earlier turns. Check the model’s multi-turn retention and context length—the longer the context, the more room you have for long-term memory.

2. The Trap You Must Avoid: the Over-Aligned “Assistant Tone”

This is where chat/companion projects fail most often.

Mainstream Instruct models are heavily safety-aligned and talk like support agents: “As an AI, I can’t…”, “Hope this helps!”—dropped into roleplay, they break character instantly.

So always test two things when selecting:

Refusal rate: how often normal story progression gets refused for no good reason;
Persona retention: after 20 turns, is it still the same character?

The community has plenty of Roleplay / Uncensored fine-tunes that companion products use as a starting point. But note: safety is your product’s responsibility. Do not rip out all safety alignment just to reduce refusals—the right move is to keep a safety floor and remove only the over-refusals that hurt the roleplay experience.

3. Current Open Candidates (as of 2026-09)

Model	License	Size	Why it fits chat
Qwen3.8-27B	Apache-2.0	27B dense, 262K context	Strong all-round, best fine-tuning ecosystem (hundreds of finetunes/adapters)—top pick
Gemma 4 31B	Apache-2.0	30.7B dense, 256K	Strong native English, native system role—easy persona control
Gemma 4 26B A4B	Apache-2.0	25.2B MoE, 3.8B active	Fast, low-cost—best value for scaling
Gemma 4 12B Unified	Apache-2.0	11.95B, 256K	Single-GPU friendly—good for a first end-to-end run/on-device
GLM-5.2 / 5.3	MIT	753B MoE	Huge weights, expensive to deploy—not worth it for small teams
Tencent Hy3	Apache-2.0	295B MoE / 21B active	Strong multi-turn intent tracking, but heavy (8 GPUs+)
Qwen3.5-397B-A17B	Apache-2.0	397B MoE / 17B active	Only if you have a cluster

How to read this table: individuals/small teams should start from Qwen3.8-27B (most stable quality and ecosystem) or Gemma 4 26B A4B (best cost). To get a single-GPU run working first, pick Gemma 4 12B. Several-hundred-billion MoE models like GLM/Hy3 are not cost-effective unless you have a cluster.

Note: models iterate fast; the table is a current snapshot. Always verify the license and fine-tuning framework support on the model card.

4. Do Not Trust Leaderboards Alone: A 3-Day Bake-Off

Leaderboards only narrow the field. The final call must use your own data. Method:

Pick 2–3 candidates (e.g., Qwen3.8-27B, Gemma 4 26B A4B, Gemma 4 31B);
Build an eval set: your real persona setup + 50–100 English conversations covering small talk, emotion, plot progression, and sensitive boundaries;
Run the same small LoRA-SFT job on each, then compare:
- Persona consistency and tone fit
- Multi-turn drift and forgetting
- Refusal rate
- Latency and per-conversation cost
Pick the winner as your base, then invest in the full data pipeline.

This bake-off takes 3 days and saves you months of rework from choosing the wrong base.

5. Decision Checklist

Tick each one when selecting a base:

License allows commercial use and self-hosting (check the license page)
Has an Instruct version and fine-tuning framework support (LLaMA-Factory / Unsloth / vLLM)
Enough community finetunes / adapters
Strong English creative writing and multi-turn quality (sub-leaderboards + real chats)
Native system prompt support
Context length meets long-term memory needs
Refusal rate and persona drift acceptable on your own eval set
Inference cost within budget (favor MoE token economics)
3-day bake-off completed—let the data decide

Summary

Choosing a base model for chat is a balancing act across four things: capability (especially roleplay), cost, fine-tunability, and ownership.

Want quality and ecosystem: Qwen3.8-27B
Want cost and scale: Gemma 4 26B A4B
Want a single-GPU start: Gemma 4 12B
Avoid: the over-aligned “assistant tone”, and giant MoE models you cannot afford to run

Remember one line: there is no “best base model”, only the one that best fits your data, budget, and gameplay—and that answer can only come from your own bake-off.