Damus
ChipTuner · 2d
nostr:npub1m3xdppkd0njmrqe2ma8a6ys39zvgp5k8u22mev8xsnqp4nh80srqhqa5sf this might be the way to organize and set pricing for locally hosted compute
ChipTuner profile picture
A "smart" decision based router unlocks better and possibly more dynamic pricing routes. The average home hoster is unlikely to be able to run more than a single usable model loaded at a time, and usually not more than a 1-3 concurrent KV caches. I don't think routerly alone solves this issue, but it facilitates a small decision model that can be tuned. It already supports round-robin queuing and a weighted routing scheme. The embedding and decision model could run somewhat local to the router and offer better per-request routing from the pool of hosters. Commercial services are appearing already of course.
liminal 🦠 · 14h
hmm not entirely sure what this library would do. But i am reminded of routstr, which allows you to offer llm services for cashu