8GB GPU 35B MoE: Host RAM and Active-Expert Cache, Not VRAM Size

Serving a 35B Mixture-of-Experts model on an 8GB GPU is not a VRAM-capacity fix. It is a host-RAM decision: keep only the active experts in the GPU cache and leave the full expert pool in host RAM. If you install based on VRAM size alone, the process may start and then stall on the first prefill or during an agent recomputation.

📑Table of Contents

1. Compare published speeds before you treat them as a pass
2. Local AI serving is a host-RAM and bandwidth problem
3. Lock one official install path
4. Translate paper tok/s into a local pass line
5. Go / no-go checklist
6. FAQ
7. Takeaway

FreeToken is an edge-native MoE serving engine from UC Berkeley and UT Austin.

• Paper: arXiv 2608.16157
• Distribution: flashml.ai plus an Apache-2.0 GitHub repo
• Discovery write-up, not the adoption test: PC Watch

Next, lock one official Desktop or Linux CLI path, write a numeric tok/s and TTFT pass line, then complete one 1919 round-trip and decide go or no-go. The comparison table, official install path, published-speed translation, and checklist below are the materials for that decision.

---

Compare published speeds before you treat them as a pass

The numbers below are author-environment measurements. They are not a measurement of your laptop. Write your own pass line first.

Published figures by setup

Setup	Published figure	Check this first	Source
8GB laptop GPU	Qwen3.6-35B-A3B NVFP4 39.3 tok/s (paper: above a 33 tok/s Codex production-trace median)	Is the checkpoint NVFP4? Is free host RAM enough?	arXiv 2608.16157
32GB gaming GPU	DeepSeek-V4-Flash 284B at 22–25 tok/s	About 13B parameters are active; the full expert pool still sits on the host	Same paper / MarkTechPost

---

Workstation and 5090 figures

Setup	Published figure	Check this first	Source
One workstation GPU	GLM-5.2 753B (40B active) 14.9 tok/s vs llama.cpp 7.3	Author hardware. Re-measure on your box	AI Daily Post
RTX 5090	Qwen3.6-35B-A3B 77–83 tok/s	Paper number. Dataconomy notes independent re-runs are still thin	Dataconomy

---

Install path at a glance

Setup	Published figure	Check this first	Source
Install path	Desktop from flashml.ai. CLI: uv pip install "freetoken[accel]". ft serve binds 127.0.0.1:1919	x86_64 Windows/Linux. macOS and AMD are roadmap	install.md / FAQ #84

---

Sources

• arXiv 2608.16157
• FlashML README
• install.md
• Dataconomy
• MarkTechPost

Figures above are as of August 2026.

---

How to read the table

Read the table as a filter. “35B on 8GB” is tied to NVFP4 and the authors’ machine. The same model in bf16 is documented against about 70 GB free host RAM. The 753B figure is a workstation GPU result, not an 8GB laptop result.

---

Local AI serving is a host-RAM and bandwidth problem

MoE cuts compute per token. It does not shrink the full expert pool. A 35B model fits in 8GB VRAM only because the design caches active experts on the GPU.

Active parameters versus host RAM

The paper’s DeepSeek-V4-Flash example activates 6 of 256 experts per layer across 43 layers. That is 13B active of 284B. The active slice fits an RTX 5090 32GB card; the pool stays on the host. Official FAQ text says the same:

• Experts live in host RAM
• Qwen3.6-35B-A3B bf16 needs about 70 GB free
• NVFP4 needs less

---

Prefill and bandwidth split

Prefill creates a different load. The union of routing can densify the expert set. MarkTechPost restates these prefill times:

• About 2s on an RTX 5090
• About 5s on PCIe 4.0 desktops
• More than 10s on a laptop x8 link

Decode misses are split by the q* policy between PCIe fill and CPU compute. That is not a static layer offload chosen at load time. Measured B_P:B_H is:

• 52.7:77.3 on an RTX 5090
• 11.8:47.5 on a 4060 laptop

---

Agent checkpoints and paper comparisons

For agents, the paper claims checkpoints at framework boundaries. Worst-case TTFT is under 44s. The paper’s comparison is:

• llama.cpp: 232s
• Ollama: 179s
• KTransformers: 946s

A community counter-result also exists: llama.cpp at 34.9 tok/s on an RTX 5060 8GB. Dataconomy states that independent verification of the speed claims is still needed.

Write free host RAM, PCIe generation, and quantization (NVFP4 / MXFP4 / BF16) before you write VRAM size. That is the actual local AI serving decision.

---

Lock one official install path

If you try it, pick Desktop or the Linux CLI first. Do not treat both as production at once.

Official install steps

1. Check the matrix. Official support is x86_64, NVIDIA Ampere (RTX 30) or newer, driver r580+ (CUDA 13). Desktop covers Windows and Linux. The pip / uv CLI is Linux. macOS, AMD, and DGX Spark (aarch64) are roadmap.
2. For Desktop, install Windows or Linux from flashml.ai. The README says that path does not need a source build.
3. For CLI, create a venv and install the accel extra. The first run needs CUDA 13 nvcc on PATH for kernel JIT.

uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"
ft --version

---

Serve, probe, and attach

4. Serve with ft serve --model ~/path/to/Qwen3.6-35B-A3B. A Hugging Face repo id is also accepted. Wait for API server is ready to serve on 127.0.0.1:1919.
5. Confirm the id with curl http://127.0.0.1:1919/v1/models, then send one turn to OpenAI /v1/chat/completions or Anthropic /v1/messages.
6. To attach a coding agent, use ft launch claude (claude / codex / dsh / hermes / openclaw / opencode). Preview with --dry-run.
7. If you will use bandwidth split, run ft bench bw once. ft serve --moe-backend auto uses fused for dense models, offload for MoE, and hybrid after a profile exists.

---

Limits and when to stop

Official limits still apply. GGUF is Gemma-4 only for now. Multimodal is text-only. GTX 1080 / 2080-class cards are out of scope; community issues also report missing kernels. Dataconomy recorded PyPI freetoken v0.1.2.

If you only have macOS, or only about 16 GB of host RAM, stop. Open a test window only when you have Windows or Linux, an RTX 30-class GPU or newer, and a realistic chance that the expert pool fits in host RAM.

---

Translate paper tok/s into a local pass line

Paper tok/s is not a reason to adopt. The real test is whether you can measure the same quantization, context, and agent load.

Restated published figures

Restated figures:

• NVFP4 35B at 39.3 tok/s on an 8GB RTX 4060 laptop, which the paper places above a 33 tok/s Codex production-trace median
• On an RTX 5090, Qwen3.6-35B-A3B BF16 is 77–83 tok/s and DeepSeek-V4-Flash MXFP4 is 22–25 tok/s, with decode within 12% of a single-turn run
• On an RTX PRO 6000, GLM-5.2 753B (40B active) is 14.9 tok/s versus llama.cpp 7.3
• Author comments quoted via PC Watch claim 3–4x decode and 6–30x prefill versus Ollama
• winkrun’s independent note uses 2–4x versus Ollama and also records the 34.9 tok/s llama.cpp counter on an RTX 5060 8GB

---

Formats and a local pass line

Supported examples include DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2. Formats include MXFP4, NVFP4, FP8, and BF16. The license is Apache-2.0. Quoted comparisons often mix quantization, context, and agent re-prefill. Keep ft bench bw plus one fixed prompt on your machine.

Write the pass line as numbers first — for example 20 tok/s for chat and TTFT under 60s for an agent. Do not copy 39.3 as the target. Whether a local LLM belongs on your desk at all is a separate open-weight versus API choice.

---

Go / no-go checklist

Do not decide from a headline or from bookmark counts. Decide from OS, GPU generation, host RAM, quantization, and a measurement method.

Checks before you install

• ✓ OS is x86_64 Windows or Linux. macOS is officially out
• ✓ GPU is RTX 30 / 40 / 50. Driver is r580+ (CUDA 13)
• ✓ Free host RAM can hold the expert pool. bf16 35B is about 70 GB. If not, size the NVFP4 / MXFP4 checkpoint first
• ✓ One quantization and one checkpoint are locked (example: nvidia/Qwen3.6-35B-A3B-NVFP4)
• ✓ One path is in use: Desktop or ft serve, not both as production
• ✓ Pass line is numeric for tok/s and TTFT. The paper’s 39.3 is not the target
• ✓ If an agent will connect, measure re-prefill after a tool call. A single chat turn is not enough
• ✓ If internal code will sit on the endpoint, define the local-only boundary and log scope first

---

Stop conditions

Stop if any of these hold: macOS or AMD only, not enough host RAM, you cannot install CUDA 13, you cannot write a pass line, or you want third-party re-runs first.

---

Next five actions

The next five actions are mechanical.

1. Copy official requirements onto one page.
2. Pick one quantization.
3. Run ft --version and one round-trip on port 1919.
4. Record tok/s and TTFT on a fixed prompt.
5. If you miss the line, compare llama.cpp or Ollama on the same prompt.

Connecting a coding agent is a second selection, not a default.

---

FAQ

Does any 35B run fast on 8GB VRAM?

No. The 39.3 tok/s figure is NVFP4 Qwen3.6-35B-A3B on the authors’ machine. bf16 assumes about 70 GB of host RAM. Do not read the parameter count alone.

---

Is this the same as llama.cpp offload?

The paper says no. It contrasts llama.cpp’s static placement at load time with FreeToken’s per-token LRU and bandwidth split. Even on 8GB, the community llama.cpp result (34.9 tok/s on an RTX 5060) is not the same setup as the paper.

---

Does official support include macOS or AMD?

Not as of the 25 August 2026 FAQ. Both are roadmap. Do not plan a trial there yet.

---

How do you attach a coding agent?

Use the OpenAI- and Anthropic-compatible API from ft serve. ft launch claude writes provider config. Preview with --dry-run. Chat tok/s does not measure re-prefill after a tool call.

---

Should you try 753B on a laptop?

No. The 753B result is a workstation GPU (RTX PRO 6000). The published 8GB laptop example is 35B.

---

Is popularity a reason to adopt?

No. Attention is only a discovery signal. Adoption is official requirements, host RAM, and a measurement on your machine.

---

Takeaway

The useful question is not “is 35B fast on 8GB.” It is whether an official path, on your host RAM and interconnect, meets a pass line you wrote in advance. Paper numbers and independent reports are cross-checks, not a substitute for a local run. Independent reproduction is still thin, a llama.cpp counter-result exists, and the v0.1.2 model matrix can move.

Do this next

Do this next, in order:

1. Match OS / GPU / RAM to the official FAQ
2. Pick Desktop or CLI
3. Lock quantization
4. Complete one 1919 round-trip
5. Record tok/s and TTFT and then go or stop

If you cannot write the pass line, do not install.

Related articles:

• 13569
• 13558
• 13573