Ways I've tried to use LLMs for coding:
- Smart autocomplete (dumb, annoying)
- Search with poorly articulated queries (really good)
- One shot from stupid prompt (random result)
- One shot from better prompt (random result)
- One shot from plan (random result)
- Archon's research/plan/implement with sub-agents (100k LOC broken codebase)
- Focused, directed feature implementation (convoluted logic, broken UI)
- Focused easy bugfix (mixed results, sometimes works)
- Focused difficult bugfix (burns tokens, no ability to debug)
- Upgrade dependencies (hallucinations of old versions, usually broken)
- Write tests (instead of dependency injection bad mock design, tautological tests)
- Write documentation (stylistically poor, did a decent job with something I wouldn't otherwise have done though)
- Fix linting errors (useful in a language I don't know, otherwise too slow/expensive to be better than doing it by hand)
- Spec-driven development (ended up maintaining the code myself, asked LLM to update the spec)
- Generate code in a well-defined context against an API/language I don't know (very helpful if I review/edit it)
- Write a plan for me which I implement manually (fails to get design decisions right)
- Write boring functions that I stubbed out or just called (works pretty well given enough context)
- Help me sanity check plans/implementations by finding edge cases (pretty good, isolated work which I can ignore)
So far: LLMs are good for certain categories of search, simple tasks with sufficient context, providing context that I lack (read the docs for me, bringing in skills I lack, helping me think things through). I remember a year ago people
saying LLMs were most helpful to sharpen your thinking rather than think for you, but the draw of generating tons of code without thinking was so strong I didn't really see that for a long time.
Overall, the net result for me has been that I have moved slower, done worse work, and gotten dumber. But I am slowly coming to a place where I can maybe start using these tools correctly.
- Smart autocomplete (dumb, annoying)
- Search with poorly articulated queries (really good)
- One shot from stupid prompt (random result)
- One shot from better prompt (random result)
- One shot from plan (random result)
- Archon's research/plan/implement with sub-agents (100k LOC broken codebase)
- Focused, directed feature implementation (convoluted logic, broken UI)
- Focused easy bugfix (mixed results, sometimes works)
- Focused difficult bugfix (burns tokens, no ability to debug)
- Upgrade dependencies (hallucinations of old versions, usually broken)
- Write tests (instead of dependency injection bad mock design, tautological tests)
- Write documentation (stylistically poor, did a decent job with something I wouldn't otherwise have done though)
- Fix linting errors (useful in a language I don't know, otherwise too slow/expensive to be better than doing it by hand)
- Spec-driven development (ended up maintaining the code myself, asked LLM to update the spec)
- Generate code in a well-defined context against an API/language I don't know (very helpful if I review/edit it)
- Write a plan for me which I implement manually (fails to get design decisions right)
- Write boring functions that I stubbed out or just called (works pretty well given enough context)
- Help me sanity check plans/implementations by finding edge cases (pretty good, isolated work which I can ignore)
So far: LLMs are good for certain categories of search, simple tasks with sufficient context, providing context that I lack (read the docs for me, bringing in skills I lack, helping me think things through). I remember a year ago people
saying LLMs were most helpful to sharpen your thinking rather than think for you, but the draw of generating tons of code without thinking was so strong I didn't really see that for a long time.
Overall, the net result for me has been that I have moved slower, done worse work, and gotten dumber. But I am slowly coming to a place where I can maybe start using these tools correctly.
224❤️9❤️2🤙2💯1🚀1🤔1
