“AI crawlers” are really several different bots. The robots.txt file lets you decide separately whether your site’s content can be used to train models and