Yeah you probably wouldn't want the KV that big, with current model runtimes. Large, full context windows start to reduce quality and performance. Usually around 80% they start to degrade in my experi...
Everything I've been researching has been pointing me towards RAG.
Which sucks because that's more stuff I have no clue about. 🤣
Which sucks because that's more stuff I have no clue about. 🤣