Menu
Article Troubleshooting

python requests module async

Narkw Brahma

Narkw Brahma

• 2 min read

Python Requests Module Async

Python's requests module is a great tool for making HTTP requests in Python. However, when making a large number of requests, it can be slow as each request is made synchronously, one after the other. This is where asynchronous programming can come in handy.

What is Asynchronous Programming?

Asynchronous programming allows multiple tasks to be executed concurrently without blocking the main program's execution. In other words, it allows a program to perform other tasks while waiting for some other task to finish. This way, a program can achieve better performance by making the most of the available system resources.

Using Async with Requests Module

To use async with the requests module, we can use the asyncio package in Python. This package provides a way to write asynchronous code using coroutines and event loops.

Here's an example of how to use async with the requests module:


import asyncio
import requests

async def make_request(url):
    response = await loop.run_in_executor(None, requests.get, url)
    return response.text

async def main():
    urls = [
        "https://www.google.com",
        "https://www.facebook.com",
        "https://www.twitter.com"
    ]

    tasks = []
    for url in urls:
        tasks.append(asyncio.ensure_future(make_request(url)))

    responses = await asyncio.gather(*tasks)

    for response in responses:
        print(response)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

In this example, we define a coroutine function called make_request that takes a URL as an argument and uses the requests module to make an HTTP GET request to that URL. The response is then returned as text.

We also define another coroutine function called main that creates a list of URLs to request and then creates a list of asyncio tasks for each URL. We then use the asyncio.gather method to run all the tasks concurrently and wait for them to complete. Finally, we print out the responses.

Other Ways to Use Async with Requests Module

Another way to use async with the requests module is to use the aiohttp package in Python. This package provides an asynchronous HTTP client and server implementation using asyncio.

Here's an example of how to use aiohttp with async:


import asyncio
import aiohttp

async def make_request(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

async def main():
    urls = [
        "https://www.google.com",
        "https://www.facebook.com",
        "https://www.twitter.com"
    ]

    tasks = []
    for url in urls:
        tasks.append(asyncio.ensure_future(make_request(url)))

    responses = await asyncio.gather(*tasks)

    for response in responses:
        print(response)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

In this example, we define a coroutine function called make_request that takes a URL as an argument and uses the aiohttp package to make an HTTP GET request to that URL. The response is then returned as text. We also define another coroutine function called main that creates a list of URLs to request and then creates a list of asyncio tasks for each URL. We then use the asyncio.gather method to run all the tasks concurrently and wait for them to complete. Finally, we print out the responses.

Share