AI Radar
Top stories
Importance 2 of 5Covered by 1 source

Open-source tool tunes cudaMemcpy to speed MoE CPU offload prefill by about 35%

A developer released a tool that tunes CUDA memory copies used when offloading mixture-of-experts layers to the CPU. On an RTX 3090 with Qwen3.6 35B-A3B in llama.cpp, prefill rose from 345 to 466 tokens per second.

Why it matters

Cheap software tuning can noticeably improve local inference on consumer GPUs for models too large for VRAM.

Timeline: llama.cpp, Qwen3.6 35B-A3B