Run small and mid-size LLMs on-premises (4B–80B)
● Measured + arithmetic — the tier where everything fits
Not every job needs a 200B-class brain. Coding assistants, triage, extraction, embeddings, customer-facing chat — the workhorse tier is 4B to 80B, and on a 128 GB unified-memory unit it fits with room to run several models side by side.
Two of the entries below are measured from our own production fleet; the rest is the same arithmetic we use to size customer deployments.
Memory fit by size class (one 128 GB unit)
| 7–9B class (4-bit) | ~4–6 GB — dozens of gigabytes to spare |
|---|---|
| 24–35B class (4-bit) | ~14–20 GB; a 35B mixture-of-experts planner holds a ~20 GB slot in our own fleet design |
| 70–80B class (4-bit) | Qwen3-Coder-Next (80B, ~3B active) — 45 GB measured on our fleet, our production coding worker |
| Device memory | 128 GB unified (one unit) |
| Typical co-residency | a coding model + a triage model + embeddings, together on ONE unit |
At 4-bit, weights take roughly half a gigabyte per billion parameters, plus context cache. The practical consequence: a single unit is a multi-model office server, not a one-model appliance.
What this tier serves well
- A private coding assistant for your engineering team — your source code never leaves the building.
- Document extraction, classification, and summarization pipelines.
- Embeddings and retrieval (RAG) serving next to the generation model.
- Internal chat and customer-service automation in your own language.
- Always-on triage and routing models that cost almost nothing to keep resident.
What we run ourselves
Qwen3-Coder-Next (80B parameters, ~3B active, 45 GB quantized — measured) is the coding worker of our own fleet: it writes and edits code for our products daily. A ~20 GB 35B-class model holds the compact-planner slot in one of our fleet's operating modes. Both figures come from our fleet records.
Honest limits
- Which exact model is best per size class changes fast; the size classes and their fit envelopes do not. We pin exact models to your use case in the assessment, not in marketing copy.
- A fine-tuned small model often beats a generic big one on a single narrow job — that trade is what our LLM Factory exists for, and we will tell you when it applies.
Frequently asked questions
- Can several models share one unit?
- Yes — that is how we run our own fleet: a ~45 GB coding model can sit next to a ~20 GB planner and smaller utility models on a single 128 GB unit, each with its own memory reservation.
- Is a small model enough for our use case?
- Often, yes — extraction, classification, routing, and focused chat rarely need a flagship. When your workload does need a bigger brain, the same unit runs a 122B-class model instead. The assessment answers this with your documents and volumes, not with a slogan.
- Can these models be fine-tuned on our data?
- Yes — this tier is the sweet spot for fine-tuning: training is affordable, and our LLM Factory benchmarks the fine-tune against leading models on your tasks before you commit, then delivers it onto your device.