Skip to content
AI & Strategy · July 13, 2026 · 8 min read

In-house AI:
why large organisations are starting to run it themselves

The quality argument for staying with expensive frontier models has expired. And of all places, government is showing what happens next.

🤖 Agent-friendly (.md)
Illustration for article: In-house AI: why large organisations are starting to run it themselves

The quality argument for staying with expensive frontier models has expired. And of all places, government is showing what happens next.

Someone who works for a Dutch municipality recently replied to a post of mine. Not with a question, but with the news that they're running their own AI on a $4,700 NVIDIA DGX Spark. A small box on a desk, no multi-year IT programme.

That struck me in two ways. So it can really be done. And, of all organisations, a municipality is ahead of most businesses I know.

TL;DR

  • Government is building its own AI, business is watching. On 3 July the cabinet chose a sovereign government cloud built on open source, while central government already runs its own AI chat that 82 percent of early users use weekly.
  • The quality argument has evaporated. The free model GLM-5.2 sits within a single point of Anthropic's flagship model — the gap between open and closed models has nearly closed.
  • Yet almost nobody switches. Convenience wins as long as nobody forces you to think about where your data goes and what you pay per token.
  • You're paying premium prices for routine work. For the bulk of your AI traffic — email triage, document processing — a local model handles it just fine.
  • In-house AI starts with an inventory, not a GPU cluster. Which part of your work is routine? That's a boardroom question, not an IT job.

Government is moving faster than business

On Saturday, the Financieel Dagblad reported that the Netherlands is scaling back its dependence on technology from China and the United States. A few days earlier, State Secretary Van der Burg sent the parliamentary letter behind that story. The cabinet has chosen its own sovereign government cloud, housed where possible in the state's own data centres. On the advice of Gartner, it picked the most far-reaching of four scenarios. The design is based on open source, which reduces vendor dependency, making the use of Microsoft Azure less of a given. You can read the coverage at Computable (opens in new window).

This isn't an abstract policy paper. Government is already running it. There's an in-house AI chat environment that runs locally in a government data centre and uses European, open-source language models. Among the first users, including staff at Rijkswaterstaat (the Dutch directorate for public works and water management), 82 percent now use the tool weekly or more, and 77 percent report a noticeable time saving on writing tasks, reports ICTMagazine (opens in new window). That's not a pilot run by three civil servants. That's a working facility where citizen data stays within its own walls.

And then there's that municipality with a Spark on a desk. Put those three things side by side and you see a movement touching every layer of government, from national ministries to the smallest municipality. Even though the cliché insists government is slow and business innovates.

The explanation is less surprising than the observation itself. A municipality isn't allowed to simply route its residents' data through an American model, so it inevitably arrives at the question of where the AI runs. Most companies don't face that same compulsion yet, so they choose the convenience of a frontier API. The sovereignty question a business can postpone for now is a precondition for government. That difference explains the head start. I wrote earlier that sovereignty isn't a model choice but an architecture choice and a governance responsibility. Government is now acting on that. Most businesses aren't yet.

The quality excuse has run out

For years there was one strong argument for sticking with the expensive models from OpenAI and Anthropic: quality. Open models were fine for hobbyists, but for serious work you fell behind. That argument held up. And it has now evaporated.

On 16 June, China's Z.ai published the numbers for GLM-5.2, an open model released under a free MIT licence that you can simply download. On a demanding benchmark for long-horizon coding tasks it scores 74.4, against 75.1 for Claude Opus 4.8, Anthropic's flagship model. On another widely used benchmark it beats GPT-5.5. All of that for roughly one-sixth of the price, reports VentureBeat (opens in new window). The gap between the best open model and the best closed model is no longer measured in generations, but in a handful of benchmark points.

I'll flag a caveat immediately: some of those figures were published by Z.ai itself and haven't been fully independently verified yet. But the direction isn't something you can explain away. Three years ago open source trailed by two years, last year by six months, and now by a handful of benchmark points.

This is where it gets interesting. You'd expect businesses to switch en masse once the quality is there. The opposite is happening. According to an analysis by Marka Development (opens in new window), the share of open models in enterprise use actually fell, from 19 to 11 percent, even as quality rose. That figure comes from a single source, so I hold it loosely, but the pattern is recognisable. It isn't a paradox. It's a pattern. As long as nobody forces you to think about where your data goes and what you pay per token, you take the path of least resistance. The quality excuse has expired, but the habit hasn't.

You're paying premium prices for toaster-level work

And that habit costs money. Because the vast majority of what runs through an AI model inside an organisation is routine: emails sorted to the right department, documents that need a handful of fields extracted, a piece of text that needs summarising or translating. That work doesn't need Opus or GPT-5.5, any more than you'd send a truck to pick up a single loaf of bread.

The hardware to do this yourself has, on top of that, become ridiculously small. A device like the NVIDIA DGX Spark, the box that municipality is using, runs models up to 200 billion parameters locally, in a housing the size of a toaster, for an MSRP of $4,699 since February's price increase, according to Tom's Hardware (opens in new window). And the models themselves are getting more efficient: a well-distilled 32-billion-parameter model today does the work a 70-billion-parameter model needed two years ago, notes Alpacked (opens in new window).

That flips the question. In-house AI doesn't start with buying a GPU cluster. It starts with an inventory of your own work: which share of our AI traffic is routine, and which share genuinely needs the heaviest models? Once you have that clear, you usually find that the bulk can move left, to a cheap or self-hosted model, and that only a small core should stay right, with the frontier. That's exactly what that municipality is doing: testing what a self-hosted model on that little box can handle, before committing to a large-scale purchase. That's not a technical trade-off. It's a governance one. It's the same question I raised earlier about staying in control of your AI costs: not less AI, but knowing where the meter runs fastest. And it connects to the distinction between the different types of AI users in your organisation, because the heavy builder needs something different from the colleague who has AI check a quick email.

Follow me on LinkedIn

Follow my updates on AI, strategy and entrepreneurship on LinkedIn

Where in-house doesn't pay off

Now the other side, because I don't want this to read like a press release. In-house AI is no miracle cure, and for many organisations it's simply more expensive. The bare price of a GPU is the floor, not the ceiling. Once you add the people who keep it running, the overcapacity for peak moments, the hours it sits idle at night, and the models you have to re-test every few weeks, the real bill climbs to three to five times that bare price. Multiple cost analyses converge on that picture, and people are consistently the largest hidden cost, not the hardware.

Then there's time. From decision to a running production environment easily takes nine to eighteen months, and during that period your organisation doesn't yet have the AI the project was meant to deliver, or it's still using the exact external models the project was supposed to replace. For a team burning through a few million tokens a day, an API is in practice often simply cheaper. Self-hosting only pays off above a clear volume threshold, or when hard requirements around privacy and oversight leave you no choice.

The answer, therefore, is rarely all-or-nothing. It's hybrid. You route the volume work to your own or a cheap model and reserve the frontier API for the small core that genuinely needs the best. Organisations that structure it this way report, according to Alpacked, savings of 40 to 70 percent compared to a stack that sends everything to the expensive models. Not by using less AI, but by routing the right work to the right place.

Small inside, big outside

Extend those lines and an architecture emerges that I summarise as small inside, big outside. A routing layer that handles daily volume with your own, open models, and only forwards the hardest few percent to an external flagship model. It sounds new, but it's a movement I've seen before, repeatedly, over twenty years. First everything sat on your own servers, then everything moved to the cloud, and now companies are pulling heavy or sensitive workloads back into their own hands because the bill and the dependency didn't hold up. For AI, that same pendulum swing isn't taking ten years. It's taking two.

What's different this time is the stakes. With ordinary software, it was about where your compute sat. With AI, it's about your data, and your data is increasingly the product itself. Whoever hands full control of their AI to a single American provider is giving away more than a technical choice. The fact that the token price you pay today is subsidised, and likely to rise in the coming years, only makes that dependency more expensive. That, too, is a reason to think now about what you want to keep in your own hands.

My claim is therefore simple. Within a few years, every large organisation will have its own AI layer, the way every company once ran its own mail server. Not because it's a political statement, but because it's cheaper and safer for the work that makes up the bulk of the load. The municipality with that Spark isn't an exception. It's ahead of the curve.

I started this piece with a message from someone who works at a municipality. What stayed with me most is that nobody there waited for permission. No years-long programme, no committee, no big budget. Someone put a small box on a desk and started testing. The question for the boardroom is therefore no longer whether it's possible. That's been answered. The question is whether you've done the one inventory that tells you which work belongs where. And that conversation doesn't start with IT. It starts with you.

Sources