User-agent: Baiduspider User-agent: Bingbot User-agent: Exabot User-agent: Yandex User-agent: * Disallow: /private/ Disallow: /assets/ Allow: /js/ Allow: /css/ Disallow: /logs/ Disallow: /scripts/ Disallow: /private-data/ Disallow: /data/ Allow: /assets/public/ Crawl-delay: 9 Disallow: /secret/ Allow: /images/ Disallow: /config/ Allow: /public/