Request to allow 百度 (Baidu) to crawl GitHub Pages

[GeeLaw]

Select Topic Area

Product Feedback

Body

Due to some historical reasons, which I cannot fully pin down, GitHub Pages currently returns HTTP 403 when 百度 (Baidu), the most popular search engine in China Mainland, crawls the website. The possible reasons (I found over the Internet) are as follows.

• Baidu crawler/spider might be making more frequent visits to GitHub Pages, affecting its performance. See this. It was a long time ago (2015).
• There was an incident of someone mounting a DDoS attack using Baidu. See this. I suspect this reason is unlikely, because GitHub allows Baidu to crawl and index its main site and it was the main site, not GitHub Pages, that was DDoS-attacked.

Rejecting indexing queries from Baidu is detrimental to the discoverability of GitHub Pages sites for the vast majority of Chinese Mainlanders (who don't know to use another search engine). In addition, the first reason (crawling frequency) is a non-issue if there is a CDN in front of GitHub Pages with custom domain --- I personally use Cloudflare and GitHub Pages server is rarely consulted even if there are many visits (98.15% of requests from the last 30 days were cached by Cloudflare).

I pledge GitHub to consider allowing Baidu to crawl Pages sites. Possible implementations:

1. Just allow it.
2. Allow it when the request is made to a custom domain.
3. Allow it when the user checks a box in their Pages site repository, with a warning that excessive access might go beyond the soft limit of 100 GB traffic per month and the advice to use a CDN.
4. Allow it when 2 or 3 is satisfied.
5. Allow it when 2 and 3 are both satisfied.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐