Gut feeling confirmed. I hear from more developers that are experimenting/working with LLMs (Large Language Models) that they are moving towards the direction I expected.

Locally running "dumb" (as in: small) models that are often combined with RAG (Retrieval-Augmented Generation) pointing at local code or doc collections. This approach is lightweight and delivers acceptable results. This domain specific approach results in far less "hallucinations" and respects privacy.