When you install WordPress, it comes with an auto generated ‘robots.txt’ file which is rather incomplete. By experience I find this is the most appropriate robots.txt file (at a minimum), for a WordPress site installed right in the root folder: User-agent: * Disallow: /feed Disallow: /*/feed Disallow: /xmlrpc Disallow: /wp- Disallow: /?p= Disallow: /*trackback Allow: […]