Menu
Article Security Check

python requests referer

Narkw Brahma

Narkw Brahma

• 1 min read

Python Requests Referrer

Python Requests library is a popular and powerful tool for sending HTTP requests in Python. It is widely used for web scraping, automation, and testing. One of the important features of HTTP requests is the Referrer header, which indicates the URL of the previous page or the source of the current request. In this article, we will learn how to set and use the Referrer header in Python Requests.

Setting Referrer Header

To set the Referrer header in Python Requests, we can use the headers parameter of the request method. Here is an example:


import requests

url = 'https://www.example.com'
referrer = 'https://www.google.com'

headers = {
    'Referer': referrer
}

response = requests.get(url, headers=headers)

print(response.content)

In this example, we are sending a GET request to a URL and specifying the Referrer header as https://www.google.com. We are passing the headers as a dictionary to the get method. The Referer key in the dictionary is case-insensitive, so we can use either 'Referer' or 'referrer'.

Using Default Referrer

If we don't set the Referrer header explicitly, Python Requests will set it to the URL of the current request by default. For example:


import requests

url = 'https://www.example.com'

response = requests.get(url)

print(response.content)

In this case, the Referrer header will be automatically set to https://www.example.com.

Multiple Referrers

Sometimes we may need to send requests from multiple referrers, such as in web scraping or testing scenarios. In such cases, we can use a session object to maintain the state and send requests with different Referrer headers. Here is an example:


import requests

url = 'https://www.example.com'
referrers = ['https://www.google.com', 'https://www.bing.com', 'https://www.yahoo.com']

session = requests.Session()

for referrer in referrers:
    headers = {
        'Referer': referrer
    }

    response = session.get(url, headers=headers)

    print(response.content)

In this example, we are sending multiple GET requests to a URL with different Referrer headers. We are using a session object to maintain the state and reuse the underlying TCP connection. This can improve the performance of our script and reduce the overhead of establishing a new connection for each request.

Conclusion

The Referrer header is an important part of HTTP requests, and Python Requests provides a convenient way to set and use it. By setting the Referrer header, we can simulate different user agents, browsers, or sources of traffic, and test our web applications more thoroughly.

Share