Yet another optimized robots.txt file for WordPress (including all the latest Googlebots)

Yet another optimized robots.txt file for Wordpress (including all the latest Googlebots)

There are a lot of self-declared "optimized" robots.txt files for WordPress around the web, but most of them don't allow full access to most Googlebots that scans our website for various reasons, such as: mobile browsers compatibility, PageSpeed insights, tag/structured data analysis, & more. Here's a list of all of them as of October 2016, including the most recent ones.

If you want to be sure that your robots.txt isn't blocking any of these, you can test your website URLs with the following tools:

If you're unhappy with the results, I strongly suggest to adopt something similar to the following:

IMPORTANT: Be sure to replace YOUR_SITEMAP_URL with your actual sitemap URL: if you're using WordPress+Jetpack, it should be something like


That's it for now: happy SEO!



About Ryan

IT Project Manager, Web Interface Architect and Lead Developer for many high-traffic web sites & services hosted in Italy and Europe. Since 2010 it's also a lead designer for many App and games for Android, iOS and Windows Phone mobile devices for a number of italian companies. Microsoft MVP for Development Technologies since 2018.

View all posts by Ryan

Leave a Reply

Your email address will not be published. Required fields are marked *

The reCAPTCHA verification period has expired. Please reload the page.

This site uses Akismet to reduce spam. Learn how your comment data is processed.