Run small and mid-size LLMs on-premises (4B–80B)
○ Memory-fit arithmetic — the tier where everything fits
Not every job needs a 200B-class brain. Coding assistants, triage, extraction, embeddings, customer-facing chat — the workhorse tier is 4B to 80B, and in 128 GB of memory it fits with room to run several models side by side.
The figures below are the same arithmetic used on every page here: parameters × bytes per parameter, plus room for context. These models run on anything from a desktop AI unit to a GPU server; we size the machine during the AI diagnostic.
Memory fit by size class (128 GB)
| 7–9B class (4-bit) | ~4–6 GB — dozens of gigabytes to spare |
|---|---|
| 24–35B class (4-bit) | ~14–20 GB |
| 70–80B class (4-bit) | ~35–45 GB |
| Reference memory | 128 GB (one machine) |
| Typical co-residency | a coding model + a triage model + embeddings, together in 128 GB |
At 4-bit, weights take roughly half a gigabyte per billion parameters, plus context cache. The practical consequence: a single 128 GB machine is a multi-model office server, not a one-model appliance.
What this tier serves well
- A private coding assistant for your engineering team — your source code never leaves the building.
- Document extraction, classification, and summarization pipelines.
- Embeddings and retrieval (RAG) serving next to the generation model.
- Internal chat and customer-service automation in your own language.
- Always-on triage and routing models that cost almost nothing to keep resident.
Honest limits
- Which exact model is best per size class changes fast; the size classes and their fit envelopes do not. We pin exact models to your use case in the assessment, not in marketing copy.
- A fine-tuned small model often beats a generic big one on a single narrow job — that trade is what our LLM Factory exists for, and we will tell you when it applies.
Frequently asked questions
Can several models share one machine?
Yes — by the arithmetic above, a ~40 GB coding model can sit next to a ~20 GB planner and smaller utility models in 128 GB of memory, each with its own memory reservation.
Is a small model enough for our use case?
Often, yes — extraction, classification, routing, and focused chat rarely need a flagship. When your workload does need a bigger brain, the same 128 GB holds a 122B-class model at 4-bit instead. The assessment answers this with your documents and volumes, not with a slogan.
Can these models be fine-tuned on our data?
Yes — this tier is the sweet spot for fine-tuning: training is affordable, and our LLM Factory benchmarks the fine-tune against leading models on your tasks before you commit, then delivers it onto your machine.
Find out which model fits your work
An AI diagnostic maps your processes, classes what is sensitive and sizes the model and the hardware against your real tasks.