Menu
Article Bugs

python requests user-agent list

Narkw Brahma

Narkw Brahma

• 1 min read

Python Requests User-Agent List

If you are working with Python requests library, you might need to set a user-agent for your requests. A user-agent is a string that identifies the client making the request to the server. Some websites might block requests that do not have a proper user-agent set. In this PAA, we will discuss how to get a list of user-agents that can be used with Python requests.

Using the "fake-useragent" Library

The easiest way to get a list of user-agents is to install the "fake-useragent" library. This library has a built-in list of user-agents that can be used with requests. Here's how to install and use the library:


# Install the library
!pip install fake-useragent

# Import necessary libraries
from fake_useragent import UserAgent
import requests

# Create a UserAgent object
user_agent = UserAgent()

# Set the user-agent in the headers of the request
headers = {'User-Agent': user_agent.random}
response = requests.get(url, headers=headers)

The "fake-useragent" library generates a random user-agent string for each request. This can be useful if you want to avoid being detected as a bot or if you want to make it look like your request is coming from a different browser or operating system.

Using a Custom User-Agent List

If you have a specific set of user-agents that you want to use, you can create your own list and randomly select from it. Here's how to do it:


import random
import requests

# Define a list of user-agents
user_agents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
    'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebkit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
]

# Select a random user-agent from the list
headers = {'User-Agent': random.choice(user_agents)}
response = requests.get(url, headers=headers)

If you decide to use a custom user-agent list, make sure that the list contains valid user-agent strings that are not outdated or invalid. An outdated or invalid user-agent can cause your request to be blocked or rejected by the server.

Share