@nprofile1q...
Making web.archive.org (Wayback Machine) as "only single service that is allowed to crawl using robots in the world" and any others commercially want to use data on Internet to purchase data from them (of course, only allowed to do ones. Would need i.e., "no AI, no commercial" option in robots.txt not to be sold to AI things) seems to be the way to go.
And force purchasing data from Wayback Machine to be outside Internet (dedicated leased line, for example) would significantly lower the unndeeded traffics.
Keeping Wayback Machine in good manner (strictly obbey robots.txt, restricting crawling frequencies, contracting with authors directly by Wayback Machine if contents are allowed to be sold, and so on) would be needed, too.
This way, all "allowed" contents that are not too often (over once a day, for example) to be updated could kept public even when the server services are gone disregarding the intentions of authors there.
Making web.archive.org (Wayback Machine) as "only single service that is allowed to crawl using robots in the world" and any others commercially want to use data on Internet to purchase data from them (of course, only allowed to do ones. Would need i.e., "no AI, no commercial" option in robots.txt not to be sold to AI things) seems to be the way to go.
And force purchasing data from Wayback Machine to be outside Internet (dedicated leased line, for example) would significantly lower the unndeeded traffics.
Keeping Wayback Machine in good manner (strictly obbey robots.txt, restricting crawling frequencies, contracting with authors directly by Wayback Machine if contents are allowed to be sold, and so on) would be needed, too.
This way, all "allowed" contents that are not too often (over once a day, for example) to be updated could kept public even when the server services are gone disregarding the intentions of authors there.