After improving my Grafana setup, I noticed thanks to the improved graphs and dashboards that my SSDs were being used way too much and the culprit was my nostr relay.
I overlooked a read-ahead option for the relay database, and it caused unnecessary reads that in turn causes unnecessary sync between SSDs (longhorn sync mechanism for replicated volumes). They were non-stop reading at 150MB/s, omg !!!
It took between 20 to 24 hours and peaked at almost 2TB, before compacting to ~950GB. The pool is mainly HDDs with SSD cache, but I don't think that the cache helped much, for this amount of data.
I did not use lightmode and did not use "--address-search".
It takes some time to adjust to the cloud native mentality as well. Some stuff that is obvious on docker powered NAS goes out the window with kubernetes.
I would use some kind of remote share if the same data should be shared across many apps.
Local storage isn't usually meant for shared data. But I can see why one want would want fast local shared data, that is something easy to do in non cloud native technologies, but it doesn't fit the cloud native mentality.