Reddit has recently revised its robots.txt file to restrict access for many search engines and AI tools, including Bing. This change, implemented on July 1, 2024, means Bing and other search engines are no longer able to crawl Reddit’s site.
According to a Microsoft representative speaking to Search Engine Land, “Bing ceased crawling Reddit after the updated robots.txt file was put into effect, which disallows all crawling of their site.”
The update to Reddit’s robots.txt file on July 1 aimed to restrict various search engines and AI tools from accessing its content, though Google was not affected. Despite earlier speculation, Google retains the ability to crawl Reddit’s site. However, Reddit’s new policy has effectively barred most other crawlers.
The issue came to light when Mark Williams-Cook observed a decline in Reddit search results on Bing, prompting further investigation by media outlets. It was later confirmed that Reddit employed IP-based detection to present different versions of its robots.txt file to search engines versus human users.
A Microsoft spokesperson confirmed to Search Engine Land that Bing complies with the robots.txt standards, stating: “Bing has stopped crawling Reddit in line with the updated robots.txt file that prohibits all site crawling.”
In a statement to The Verge, Reddit spokesperson Tim Rathschmidt clarified that the site’s actions are not connected to its recent agreement with Google. He explained, “We have been negotiating with several search engines but have been unable to reach agreements with all, as some could not or would not make enforceable commitments regarding the use of Reddit content, including for AI purposes.”
The significance of this move lies in Reddit’s strategic leverage following its licensing agreement with Google. This arrangement enables Reddit to impose restrictions on other search engines and AI tools, while Google continues to drive substantial traffic to Reddit and even experiments with enhanced search result features.
This development raises questions about whether other major websites might adopt similar tactics, potentially impacting smaller publishers and content creators.
In the meantime, users can expect limited new Reddit content to appear in Bing search results.
You Might Be Interested In
