Yet another optimized robots.txt file for WordPress (including all the latest Googlebots)

Yet another optimized robots.txt file for Wordpress (including all the latest Googlebots)

There are a lot of self-declared “optimized” robots.txt files for WordPress around the web, but most of them don’t allow full access to most Googlebots that scans our website for various reasons, such as: mobile browsers compatibility, PageSpeed insights, tag/structured data analysis, & more. Here’s a list of all of them as of October 2016, including the most recent ones.

If you want to be sure that your robots.txt isn’t blocking any of these, you can test your website URLs with the following tools:

If you’re unhappy with the results, I strongly suggest to adopt something similar to the following:

IMPORTANT: Be sure to replace YOUR_SITEMAP_URL with your actual sitemap URL: if you’re using WordPress+Jetpack, it should be something like www.yoursite.com/sitemap.xml.

 

That’s it for now: happy SEO!

 

 

RELATED POSTS

About Ryan

IT Project Manager, Web Interface Architect and Lead Developer for many high-traffic web sites & services hosted in Italy and Europe. Since 2010 it's also a lead designer for many App and games for Android, iOS and Windows Phone mobile devices for a number of italian companies.

View all posts by Ryan