Menu
Article Performance Hacks

make python requests faster

Narkw Brahma

Narkw Brahma

• 2 min read

How to Make Python Requests Faster?

If you're working with Python requests library, you might have noticed that it can take a long time to retrieve data from a web server. This can be frustrating if you're dealing with large amounts of data or need to make multiple requests in a short amount of time. Fortunately, there are several ways to speed up your Python requests.

1. Use Connection Pooling

By default, the requests library uses a new TCP connection for every request. This can be slow because establishing a new connection takes time. Connection pooling allows you to reuse existing connections, which can significantly speed up your requests.


import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

retry_strategy = Retry(
    total=5,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "OPTIONS"]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)

response = http.get("https://www.example.com")

2. Use Async Requests

If you need to make multiple requests in parallel, consider using asynchronous requests. This allows you to send multiple requests at the same time, which can be much faster than sending them sequentially.


import asyncio
import aiohttp

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = []
        for url in urls:
            task = asyncio.ensure_future(fetch(session, url))
            tasks.append(task)
        responses = await asyncio.gather(*tasks)
        for response in responses:
            print(response)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

3. Use a Faster Serializer

If you're sending data in a request body, consider using a faster serializer. The default JSON serializer in Python can be slow for large amounts of data. There are several faster alternatives, such as ujson and msgpack.


import ujson

data = {'foo': 'bar'}
response = requests.post(url, data=ujson.dumps(data), headers={'Content-Type': 'application/json'})

4. Use a Faster DNS Resolver

When you make a request, your computer needs to resolve the domain name to an IP address. This can be slow if you're using the default DNS resolver. Consider using a faster resolver, such as Google's Public DNS.


import requests
import socket

def dns_resolver(host):
    return socket.gethostbyname(host)

response = requests.get('https://www.example.com', dns_cache=dns_resolver)

These are just a few ways to speed up your Python requests. Depending on your specific use case, other solutions may be more appropriate. Keep experimenting and testing to find the best solution for your needs!

Share