Xinference

Xinference, from the Xorbits project, is a server for running open models yourself: it starts language, speech-recognition and multimodal models with a single command, keeps them behind an interface that OpenAI clients already understand — including function calling — and adds a command line, a remote procedure interface and a web console for managing what is loaded. A model catalogue supplies the weights so you do not have to find them, and it will use graphics cards and processors together rather than insisting on one kind of hardware. Recent work in the repository adds automatic batching of concurrent requests, distributed inference across workers, and a version that reconfigures a running model instead of forcing a restart.

It suits a team that wants an internal model platform rather than a single local binary: several models of different kinds available to several projects, deployable on your own machines or in a private cloud, with an enterprise edition sold separately. It suits less well one person on one laptop who wants the smallest possible way to run a model, where a single local runner is less to install and less to keep alive. It also suits less well anyone unwilling to take on the operation of a service: a cluster of workers is a service, with the deployment and monitoring that implies.

Not verified: nothing was installed and no model was loaded, so throughput, the batching behaviour and the distributed mode were not exercised. The feature list and the release details come from the repository's own description; the repository interface reported about 9,608 stars, 875 forks, the Apache-2.0 licence and a push on 11 October 2026. The documentation site and the enterprise edition's pages were not opened, so the difference between the free edition and the paid one, and any pricing, are unknown. Whether the local runners match the fastest single-machine engines for the same model was not measured, and the claim that hardware of different kinds is used together is the project's own.