Damus
Aldin profile picture
Aldin
@Aldin
There is this tech called intel optane. It is a fast memory storage ment to be used in RAID with an HDD. In that setup, it serves for caching, making slow HDD feel fast. It is about 10 years old; since abandoned because NVMe SSDs caught up.

I have this in my old-ish gaming computer which I plan to repurpose. I am thinking of experimenting local LLMs on the nvidia graphics card, or other stuff. The point is to learn.

I could also use the machine for other home lab/server things. I am thinking about putting btc node on it, monero etc. This is where optane comes to play. Because full nodes read and write to the disk all the time, I maybe able to releave some of it with a proper optane setup. Obviously not with HDD, but with my 1TB SATA SSD instead. Node may be able to sync faster with it, or at least save some read/write cycles.

Would anyone be interested in reading about results of this kind of experiment? I do not know, maybe this idea is not worth much.
6
m4d4m · 1w
Afaik it won't help much because if model must be split between VRAM <> anything else - it's slow. And there is the KV cache for the context adding to the load. Iirc the inference, in case it doesn't using weights from VRAM, goes for RAM but this is computed on CPU - it's slow as hell (also dependen...