Diacone Frost · 10w I think that would quickly endup in an 60gb index file 🤔 I use this scheme for years (kappa architecture): stream - kafka - indexers (like opensearch or adhoc/specialies processors) - kafka - app... Vitor Pamplona @VitorPamplona 1765389116 Maybr symbolic links for some of the indexing? We can also hashtable IDs to minimize size of the index