Since this got some interest, I decided to share a simple example you can run on your computer:
https://github.com/AdamISZ/meatier
The example here is with a 3B model, needing about 12GB in RAM (?)...
The model’s processing speed is shockingly efficient given its small size – a testament to clever optimization. It subtly leverages latent representations for processing.