llm-d 1.0 arrives, the open software for running AI models on in-house servers
IA en un minuto newsroom · Editor: Jon Elgezabal
In 30 seconds
llm-d, the open project driven mainly by Red Hat, IBM and Google, has released version 1.0 for serving AI models on each company's own machines on top of Kubernetes. The new edition focuses on daily operation: priorities between customers, upgrades without downtime and turns that return to the machine holding their context. In one demonstration, 3,000 agents came out between five and ten times cheaper than the same model bought as a service.
The people in charge of llm-d, an open-source project for running artificial intelligence models on each organization's own infrastructure, have presented version 1.0 at PyTorch Conference North America, as IBM Research explained on its blog. Red Hat, IBM and Google are its lead maintainers, and they founded it last year together with NVIDIA and CoreWeave; since then, Mistral AI, Lambda, Intel, Hugging Face, Cisco and AMD have joined as contributors. The Cloud Native Computing Foundation (CNCF) hosts it as a sandbox project.
llm-d runs on Kubernetes (a system for managing applications spread across many servers) and, according to IBM, lets organizations deploy any model, with any chip and in any cloud. It does not replace the engines that execute the models, such as vLLM or SGLang: it works alongside them and adds the spreading of requests across machines and what is needed to keep them running as a service. IBM describes it as an alternative that does not tie a company to a single model, cloud or hardware vendor.
Since its launch, the project has relied on five techniques: sending each request to the machine that may already hold its context in memory, moving that memory between GPU, CPU and storage so it lasts longer, grouping jobs to process them better, splitting the reading of the prompt from the generation of the answer, and spreading large "mixture of experts" models over many accelerators. IBM points out that none of them is new in this version; what is new is how they fit together and the maturity to operate them.
Version 1.0 is designed for the daily life of a service that is already up: it gives some requests priority over others when several customers share the same machines, so that bursts of work do not slow down services that need fast answers; it allows upgrades without interrupting the service; and it keeps multi-turn conversations on the server that already holds their context, so it does not have to be processed again. Carlos Costa, a Distinguished Engineer at IBM Research and co-founder of the project, sums up what this version adds as a stable API, several customers with different priorities on the same infrastructure, and upgrades without downtime. According to IBM, this routing is especially useful for AI agents, which keep coming back to the same code, the same documents and the same instructions.
IBM gives two figures. In a recent test with coding agents, llm-d's routing system made use of 97% of the already processed context that could be reused. In another demonstration, IBM and Red Hat had 3,000 software engineering agents working at the same time on in-house servers for 5 to 10 times less than it costs to use the same open model as a service. The announcement does not say which model was used in those tests.
Version 1.0.0 is already published on GitHub; the previous one, 0.10.0, came out on September 29.
Why it matters · analysis and opinion
Until now, the debate about serving models in-house revolved around the speed of each technique on its own; this version shifts the focus to something less flashy and more decisive: keeping the service up when people and agents share it and it has to be updated without stopping it. For teams that pay every month for calls to someone else's models, it opens an alternative built from open parts and backed by large companies. It is not for everyone: it requires handling Kubernetes, owning or renting machines with accelerators and choosing an open model that is up to the job. Also, the announced savings come from a demonstration by IBM and Red Hat, which lead the project, with no independent measurement. Anyone who already runs their own infrastructure can try it on a specific workload and compare bills before moving anything important.
Official source: IBM Research · Written with the help of AI: how we make the news
