Clients

Clients contains instruments that are suited to make requests during scraping.

It includes following clients:

• HttpClient: to fetch web pages or files
• FtpClient: to fetch files from ftp
• TorClient: to proxy client requests via tor
• Proxy6Client: to proxy client request via any of proxy6 proxies
• ProxyListClient: to proxy client request via any of the proxies in the list in /tmp/clients_proxy_list.txt
• ProxyList: to select proxy client based on CLIENTS_PROXY_CLIENT variable (e.g. list or proxy6)

It also implements a special wrapper around of HttpClient:

• Recaptcha::Client: to visit websites behind recaptcha blocks

Important ENV variables:

• CLIENTS_PROXY_CLIENT: to control which proxy client will be selected by ProxyClient dispatcher (valid values: list or proxy6)
• PROXY6_KEY: API key for proxy6.net service
• CAPTCHA_SOLVER_KEY: API key for 2captcha.com service
• TOR_PORT: Base port for tor SOCKS5 proxy
• TOR_CONTROL_PORT: Base port for tor controls
• HTTP_TOR_PORT: Base port for http middleman proxy for TorClient (e.g. polipo)