configure proxy in python requests

How to Configure Proxy in Python Requests

If you are working on a project that involves web scraping or data crawling, you might need to use proxy servers to avoid getting blocked by websites. In Python, you can use the requests module to send HTTP/HTTPS requests to websites. In this tutorial, we will discuss how to configure proxies in Python Requests.

What is a Proxy?

A proxy is an intermediary server that acts as a gateway between your computer and the internet. When you request a web page or resource, the proxy server retrieves it on your behalf and sends it back to you. Using a proxy can help you hide your IP address, bypass internet censorship, and improve your online privacy.

How to Set Proxy in Python Requests

Python Requests has built-in support for proxies. You can specify the proxy server URL using the proxies parameter when sending requests. Here is an example code snippet:

import requests

proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080'
}

response = requests.get('https://www.example.com', proxies=proxies)

print(response.content)

In the above example, we have defined two proxies: one for HTTP requests and another for HTTPS requests. You can customize the proxy URL according to your requirements.

Proxy Authentication

If your proxy server requires authentication, you can pass the username and password as part of the proxy URL. Here is an example:

import requests

proxies = {
    'http': 'http://user:[email protected]:3128'
}

response = requests.get('https://www.example.com', proxies=proxies)

print(response.content)

In the above example, we have included the username and password in the proxy URL. You can replace them with your own credentials.

Using Environment Variables

If you don't want to hardcode the proxy URL in your code, you can use environment variables to store the proxy configuration. For example, you can set the HTTP_PROXY and HTTPS_PROXY environment variables to the proxy URL. Here is an example:

import requests
import os

proxies = {
    'http': os.environ.get('HTTP_PROXY'),
    'https': os.environ.get('HTTPS_PROXY')
}

response = requests.get('https://www.example.com', proxies=proxies)

print(response.content)

In the above example, we have used the os.environ.get() method to get the values of HTTP_PROXY and HTTPS_PROXY environment variables. If these environment variables are not set, the get() method will return None.

Conclusion

Configuring proxies in Python Requests is a simple process. You can specify the proxy URL using the proxies parameter, or use environment variables to store the proxy configuration. If your proxy server requires authentication, you can pass the username and password as part of the proxy URL.