A bit tired of clunkyness of Ollama+AnythingLLM, I decided to try something new.

#LocalAI is a great piece of software. All-in-one solution for downloading models, backends, WebUI for interacting with the AI, and all of that running in Docker with GPU acceleration.

I even used "Configurable Button" Plasmoid to have an on/off switch on my panel, so I'm one click away from starting a local AI model.

And the performance! 95 tok/s on RX7900XTX

#ai #local #docker #homelab #pc #gpu #plasma