Stockholm, Sweden / Engineering
Running LLMs on-premises in 2026: what actually works
By Markus Lykonhold, founder of Kaldryn
19 July 2026 · 9 min read
The 2026 baseline stack
An open-weight model, quantized to FP8 or FP4, served by vLLM on one or a few Blackwell-class GPUs, with retrieval over your own documents. That setup handles most enterprise workloads, fits in a single rack unit or under a desk, and your data never leaves the building.
Running a large language model inside your own building went from a research project to a procurement decision. Kaldryn One exists because of that shift, and building it means my team lives in this stack every day. This post describes what we actually run, what the hardware costs, and where deployments fail. It is written from the workshop floor, not from a market report.
The models are no longer the bottleneck
Open-weight models have closed most of the gap with proprietary APIs on the tasks enterprises actually buy: summarization, drafting, extraction, retrieval-augmented answers, and coding assistance. Mixture of Experts is now the dominant architecture for large open models, which is why a 400 billion parameter class model can run on hardware that could never hold a dense model of that size.
What we look at when shortlisting a model for a customer: task fit, language coverage (Swedish matters in our market and quality varies wildly), license terms, and VRAM budget. The specific model names rotate every quarter, which is itself the point: when weights are open, you can swap the model without swapping the vendor.
Hardware that actually ships
The table below is the honest 2026 landscape as we see it when speccing deployments.
| Tier | Example hardware | Memory | What it runs |
|---|---|---|---|
| Desk-side | NVIDIA DGX Spark, about 4,700 dollars | 128 GB unified | 70B-class models at 8-bit, small team workloads |
| Single production GPU | RTX PRO 6000 Blackwell | 96 GB GDDR7 | 70B at FP8 with headroom, high-throughput RAG |
| Small cluster | 4x RTX PRO 6000 | 384 GB combined | Large MoE models, multi-team concurrency |
Reference points: the DGX Spark listed at 4,699 dollars in early 2026 and holds a 70B model at 8-bit quantization in its 128 GB of unified memory (Digital Applied). A 70B model needs roughly 35 to 40 GB of VRAM at 4-bit, about 70 GB at FP8, and about 140 GB at FP16 (VRLA Tech).
Quantization: the free lunch that is almost real
FP8 halves memory use with minimal quality loss on most workloads, and native FP4 support on Blackwell hardware pushes further: a 70B model fits in about 35 GB instead of 140 GB at full precision (VRLA Tech). We ship FP8 by default.
Where quality actually degrades: very long context, low-resource languages, and edge-case reasoning. We test Swedish specifically before and after quantization, because a model that is fine in English can get noticeably worse in a smaller language once compressed.
Serving: vLLM won, and that is fine
vLLM became the de facto production standard for good reasons: paged attention, continuous batching, and tensor parallelism that actually works. Most deployments need surprisingly little tuning beyond tensor-parallel size and the quantization flag. The part that needs real engineering is everything around it: model lifecycle, evaluation, monitoring, and updates. That surrounding layer is most of what Kaldryn One automates.
What breaks in production
The uncomfortable truth about self-hosting is that the GPU is the cheap part. A self-hosted deployment realistically eats 10 to 20 engineering hours per month for maintenance and monitoring, which is 750 to 3,000 dollars monthly at senior rates (SitePoint), and real all-in costs typically land at 3 to 5 times the raw GPU price (GigaGPU).
The three things everyone underestimates:
- Model updates. New weights ship monthly, and every swap needs regression evaluation against your actual workloads, not a leaderboard.
- Evaluation. Without a fixed eval suite you cannot tell an upgrade from a regression.
- Backup and recovery. An appliance has to survive a disk failure without a data scientist on site.
If you are weighing this workload against just paying per token, I wrote up the break-even math on self-hosting separately.
Compliance became a runtime feature this year
On August 2, 2026, the EU AI Act reaches its next enforcement stage: national market surveillance authorities gain full investigation and enforcement powers, transparency obligations under Article 50 apply, and general-purpose AI obligations become enforceable with fines up to 15 million euros or 3 percent of global turnover (Kennedys). The Digital Omnibus signed on July 8, 2026 deferred the Annex III high-risk obligations to December 2, 2027 (Council of the EU), but the direction is set.
Running on-premises does not exempt you from the AI Act. What it does is collapse the data-transfer chapter of every impact assessment, which in practice is the chapter that kills projects.
What I would deploy this month
- Small team, mostly document Q&A: one desk-side unit, a 20 to 30B model at FP8, local retrieval. Do not overbuy.
- Mid-size company, mixed workloads: one 96 GB class GPU, a 70B model at FP8, proper eval suite from day one.
- Regulated enterprise, many teams: a small cluster and a large MoE model, or frankly, an appliance with the operations already packaged. That is what we build, and the reason we built an appliance at all.
None of this is exotic anymore. The models are open, the serving stack is boring in the best way, and the hardware fits under a desk. The hard part has moved to evaluation, updates, and operations. Plan for that part first.