User-agent:

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0

robots.txt

curl -A

See also

• Scraping, JA3, User-agent, Scrapfly
• robots.txt, ads.txt, User-agent:
• cURL, curl -s, -X, -x, curl -T, curl --help, curl -L, curl -m, curl -o, curl -X POST, curl -d, curl --header, curl -I, curl --silent, curl -Is, curl -k or curl --insecure, curl -XPUT, curl -d, ~/.curlrc, curl -i, curl -w, curl -vvv, curl --version, curl --help