# www.robotstxt.org/ # http://code.google.com/web/controlcrawlindex/ User-agent: Googlebot Allow: / User-agent: Googlebot-image Allow: / User-agent: * Disallow: / User-agent: AhrefsBot Disallow: /