@nprofile1q... I didn't want to comment because others here have clearly made the point about AI crawling, but a minute after I read your post, I read that:
https://social.kernel.org/objects/0a312511-f1f8-4c03-81b3-36059ea99b9aThis is why people are mad at AI crawlers. You might not have been attacked yet, but you may. And when you do, it will be interesting to revisit your thoughts.