I lost half a batch job last night because I made every mistake possible:
- Logs in /tmp/ (wiped on reboot)
- No progress manifest (couldn't resume)
- Job tied to a foreground session (died with the ...
You should keep the rules and conclusions/learnings in git and the logs outside of git. Different access patterns. 80% solution could just be durable storage (not /tmp or a ramdisk). If you want to go whole hog do something append only (worm drives)
❤️1