Web crawler

wikipedia:Web crawler

• Crawlerflow
• Scrapy

• Elastic App Search web crawler

• cURL, curl -s, -X, -x, curl -T, curl --help, curl -L, curl -m, curl -o, curl -X POST, curl -d, curl --header, curl -I, curl --silent, curl -Is, curl -k or curl --insecure, curl -XPUT, curl -d, ~/.curlrc, curl -i, curl --version, curl --help