Robots exclusion standard (robots.txt)

wikipedia:robots.txt

Elastic App Search web crawler

Failed to fetch robots.txt: SSL certificate chain is invalid [unable to find valid certification path to requested target]. Make sure your SSL certificate chain is correct. For self-signed certificates or certificates signed with unknown certificate authorities, you can add your signing certificate to Enterprise Search Crawler configuration. Alternatively, you can disable SSL certificate validation (non-production environments only).

User-agent:

• nofollow and robots.txt policies.

Related

• Elastic App Search web crawler
• https://en.wikipedia.org/robots.txt
• https://dubai.dubizzle.com/robots.txt
• wget -e robots=off --mirror https://www.mywebsite.org

See also

• robots.txt, ads.txt