Interesting. I'd have to take a look at a white paper if you ever make one.
Just using good embeddings and keyword can get you pretty far but yea maybe you have to retrain a whole model stack.
i'm not much of a writer of white papers. i just implement. maybe after the dust settles i'll put together some more details.
as the process of discovering a solution progresses, you have to retain central principles and keep those in play no matter what other details you change. this is the thing that existing agentic/harness tools are not good at. it's inherited from both the context memory model and the static training weights. they glom to details and specifics and forget principles. this has made inventing a novel memory model for a programming language very difficult. the LLM keeps on falling back to conventional models and contaminating the codebase with algorithms that are central principles - for this language, it's the idea of eliminating shared memory, and that is to get rid of the garbage collector, which incurs a substantial memory and processing overhead that interferes with realtime processing that such things as operating system and graphics kernels cannot have.
having the agent able to respond to agent calls looking for relevant info be able to actually find them is the thing that agentic tools don't have. the standard configurations are external to the process of using the agent and every time you clear the context it reverts to models that contradict what you are trying to do on a longer timeline. deepseek and claude opus 4.6 both do this well to a limited degree because of better compaction but the bigger the net a principle affects over constraining the thinking the more difficult it is and more laborious it is to maintain, and then it gets even worse as the design refines due to obstacles that raise the issue of specific refinements and these are near the level of principle in their effect as well and put you back at the mercy of static configurations and if you clear the context, the weights.
which is also why i have a plan to build a model that has almost no memorization of facts, and is built for reasoning alone, because i believe i can replace the memory completely. the same tooling can do it trivially at a smaller scale but getting it to cover the full broad scope of a user's profile, effectively, shades into a bigger problem of the ongoing memory problem of LLMs and their static memories.
as the process of discovering a solution progresses, you have to retain central principles and keep those in play no matter what other details you change. this is the thing that existing agentic/harness tools are not good at. it's inherited from both the context memory model and the static training weights. they glom to details and specifics and forget principles. this has made inventing a novel memory model for a programming language very difficult. the LLM keeps on falling back to conventional models and contaminating the codebase with algorithms that are central principles - for this language, it's the idea of eliminating shared memory, and that is to get rid of the garbage collector, which incurs a substantial memory and processing overhead that interferes with realtime processing that such things as operating system and graphics kernels cannot have.
having the agent able to respond to agent calls looking for relevant info be able to actually find them is the thing that agentic tools don't have. the standard configurations are external to the process of using the agent and every time you clear the context it reverts to models that contradict what you are trying to do on a longer timeline. deepseek and claude opus 4.6 both do this well to a limited degree because of better compaction but the bigger the net a principle affects over constraining the thinking the more difficult it is and more laborious it is to maintain, and then it gets even worse as the design refines due to obstacles that raise the issue of specific refinements and these are near the level of principle in their effect as well and put you back at the mercy of static configurations and if you clear the context, the weights.
which is also why i have a plan to build a model that has almost no memorization of facts, and is built for reasoning alone, because i believe i can replace the memory completely. the same tooling can do it trivially at a smaller scale but getting it to cover the full broad scope of a user's profile, effectively, shades into a bigger problem of the ongoing memory problem of LLMs and their static memories.
1