Running a 35-billion-parameter model on a single consumer graphics card usually means compromises. A new open source project called FastLocalAI tries to keep those compromises small. It's a tuned setup for running Alibaba's Qwen3.6-35B-A3B on an NVIDIA GPU with just 16 GB of VRAM, paired with the OpenCode coding agent.
The project is available on GitHub at https://github.com/24high/FastLocalAI. It uses llama.cpp's llama-server and moves part of the model into regular system memory so the rest fits on the card.
Why the model doesn't fit and how it works anyway
Qwen3.6-35B-A3B is a Mixture-of-Experts model. The quantized Q4_K_M version from Unsloth, which the project uses by default, is about 22 GB. That is too large for a 16 GB card.
Every large language model works with a fixed amount of text at a time. This limit is called the context window. It includes everything the model can see in a single exchange: your instructions, any documents you share, earlier messages in the conversation, and the reply it is writing.
Measured in tokens
Context windows are measured in tokens, not words. A token is a small piece of text, often a short word or part of a longer one. When a conversation grows past the limit, the oldest material drops out of view or has to be shortened. The model does not keep it anywhere else. For that exchange, it is simply gone.
Open-weight models are improving quickly, and that progress is changing how organizations approach AI. A growing number of companies are no longer relying only on externally hosted services. Instead, they are running smaller models on their own infrastructure.
Privacy and Cost Drive the Shift
Two motivations stand out. The first is privacy. When a model runs on a company's own hardware, sensitive data stays inside its environment. The second is cost. For many workloads, a compact self-hosted model can be a more economical choice than paying for access to a larger external system.
CyrioX is financed by advertising. You can choose how you want to use this website:
With advertising: we load an advertising script from a third-party ad network. The ad network may set cookies, use your IP address and device information, and may process data outside the EU.
Ad-free for €0.99 per month: no advertising and no advertising tracking. Cancel at any time.
You can change your decision at any time via "Cookie Settings" at the bottom of every page.
