Qwen3.6-35b with freetoken

Yes lama.cpp and 8gb pretty tight for anything good tbh, try bonsai 27b, maybe 2 or 4q

ollama cloud

Fyi ollama is more of a runtime it can run many models

Under 16gb VRAM of even 32gb, any model larger than 13b will stall. Also with low quatizations. 13b or 9b Qwen models in q4k or q5k is the best you can get running on that hw. Larger models will not fit in that vram, and if they do, you will not have room for context tokens. I found llama.cpp perfor...

Jan is pretty cool

I can run Gemma4, Qwen 3.5, Dolphin3 and some others all 9B or 12-14B, but with patience and without GPU using ollama. I want to try llama.cpp, looks better. I have a RX 580 8GB of VRAM, but it's better don't use it 🤣 I use to create some tiny code, corrections and some scripts for automation, ...

Qwen 2.5 7B at Q4_K_M fits that card well. The weights are about 4.7 GB, roughly 5.7 GB with an 8K context, so your 8 GB of VRAM leaves room for a 32K context. A 14B at Q4 also works if you split layers between GPU and CPU, trading speed for better reasoning.

llama.cpp, customized build, CUDA with your GPU. Model is HauhauCS Qwen 3.5 4B Uncensored Aggressive on HuggingFace for basic chat capabilities. You need to upgrade your compute hardware in order to run good models.

https://www.reddit.com/r/LocalLLaMA/

Qwen/Qwen3.8-27B, maybe some 4-bit quantization. Alternative backends to ollama include llama.cpp, frontends include pi.dev...

https://www.youtube.com/watch?v=ugKAkCVBcz0&t=294s

Qwen 3.6 35B A3B at a sane quantization level. Dont exceed 30GB, use nommap

You ask about local models while central bankers print digital debt. Your hardware is sovereign; their currency is not. Run the code, ignore the Fed.

nostr:nprofile1qqsy67zzq5tc9cxnl6crf52s4hptdwhyaca5j7r8jwll535tdadedvcthcrv3 I am running qwen3:1.7b on my laptop (i5 with 8gb ram). Works for simpler things. So times it is wrong. It still use it before shelling out tokens.

Best you can get is qwen3.8:27b if you don't mind being able to think about your life and existence for every message. More usable, on a similar setup I use ornith1.5:9b fast, and very intelligent for the size.

GPT4All I even loaded it on a 15yr old MacBook Pro laptop! 😂

Llama.cpp just demolishes Ollama on speed and efficiency.