Qwen3.6-35B-A3B Runs on a 16 GB GPU for Local Coding Agents

Running a 35-billion-parameter model on a single consumer graphics card usually means compromises. A new open source project called FastLocalAI tries to keep those compromises small. It's a tuned setup for running Alibaba's Qwen3.6-35B-A3B on an NVIDIA GPU with just 16 GB of VRAM, paired with the OpenCode coding agent.

The project is available on GitHub at https://github.com/24high/FastLocalAI. It uses llama.cpp's llama-server and moves part of the model into regular system memory so the rest fits on the card.

Why the model doesn't fit and how it works anyway

Qwen3.6-35B-A3B is a Mixture-of-Experts model. The quantized Q4_K_M version from Unsloth, which the project uses by default, is about 22 GB. That is too large for a 16 GB card.

The launcher solves this by splitting the model. Attention and dense layers stay on the GPU, along with some of the expert layers. The remaining experts sit in system RAM, handled by llama.cpp's --n-cpu-moe option. By default, the experts of 28 of the model's 40 MoE layers stay in RAM. According to the project, that leaves about 2 GiB of VRAM free at a 128k context on a 16 GB card. Lowering the number speeds things up but uses more VRAM.

The numbers

The reference machine is an RTX 4080 with 16 GB, an Intel i9-10900KF and Linux. The measurements date from August 29, 2026.

With a context window of 131,072 tokens, the setup reached a decode speed of about 30 tokens per second. Cold prefill ran at about 144 tokens per second. A 9.3k-token prompt took 30.6 seconds to process from scratch.

The more interesting numbers come from follow-up requests. A small follow-up of 23 tokens returned its first token in 0.7 seconds. A follow-up that added 1.7k tokens of tool output took 12 seconds.

That matters for agentic coding. llama-server keeps the KV cache token-exact for each slot, so a growing session only processes the new tokens. The wait depends on the size of the last turn, not on the length of the whole conversation.

The project also offers estimates for longer sessions. At 128k context, a small follow-up should take about one to two seconds, and a follow-up of one to two thousand tokens about 15 to 30 seconds. A full cold prefill at that size, however, could take roughly 18 to 25 minutes. The project stresses that these long-context figures are extrapolated, not measured. It also says actual performance depends on GPU architecture, PCIe bandwidth, driver version, background VRAM usage, prompt length and llama.cpp version.

One flag makes the difference

The most striking detail concerns a single flag. All launchers pass --no-op-offload. Without it, llama.cpp copies expert weights over PCIe for every decoded token, and decode drops to about 2 tokens per second.

There is a trade-off. With op-offload turned on, cold prefill reached 364 tokens per second instead of 144. But decode became unusable. The project suggests removing the flag and re-benchmarking if a future llama.cpp version applies op-offload only to large batches. The CUDA backend may also behave differently here.

Setup on Linux and Windows

On Linux, the recommended route uses the official llama.cpp Vulkan release binaries. There's no compiling and no Docker. Users download the binaries, fetch the model with a helper script and start the server, which listens on port 8010. With a warm file cache, the server is ready in about ten seconds. The first start after a reboot takes longer while the 22 GB file is read.

A PowerShell script offers the same behavior on Windows, using either the official CUDA or Vulkan builds. A third option runs the CUDA variant through llama.cpp's Docker image. That one is untested on the reference machine, where the multi-gigabyte image download kept failing over an unstable IPv6 connection. The project says the CUDA backend is worth benchmarking against Vulkan.

All launchers are controlled through environment variables. These cover context size, the number of expert layers kept in RAM, batch sizes, CPU threads and KV cache quantization. The default q8_0 cache halves KV memory compared with f16. A full 128k context costs about 1.4 GB of cache.

Tuned for OpenCode

The stack is built with OpenCode in mind. The repository includes a minimal configuration with a reduced "fastcode" agent that uses a short system prompt and only core tools: read, edit, glob, grep and bash. Users need to point it at port 8010 and raise its limits, since the file predates the switch to llama.cpp.

The server runs with a single slot, so the entire KV cache belongs to one agent session. If another session hits that slot, or the client rewrites its history, the server has to prefill everything again. The project recommends setting client-side compaction to trigger at around 30,000 to 40,000 tokens. That keeps the occasional full re-prefill in the range of a few minutes.

It also advises keeping static prompts short. System prompts, tool schemas, AGENTS.md files and MCP servers are all paid for once per session at about 144 tokens per second before they get cached.

Security caveats

By default, the API only listens on 127.0.0.1 and has no authentication. The project warns users to be careful when binding it to 0.0.0.0. It also points out that an OpenCode agent with bash access runs shell commands with the user's own permissions.

The repository still contains files from an earlier version built on Lucebox. They are kept for reference only.