ABH3PO on nostr

Open in Damus

Im getting 50tk/s with qwen3.6 35B FP8 on vllm with speculative decoding. 250k tk context window. Have to do a lot of configs to get things working right

ABH3PO @ABH3PO 1781926609

Thats considerably lower than what I would have thought especially for a 35b model, do >70b models run at similar speeds?

1🧡1

It depends on the number of active parameters. Qwen3.6 27B (dense) was running at 35-40tk/sec while qwen3-coder-next (moe 80B 3A) was running at 45tk/sec. Dm me and i’ll share you a url and API key you can use to test them out on my DGX Spark!