Menu
Article Pro Tips

python requests async response

Narkw Brahma

Narkw Brahma

• 2 min read

Python Requests Async Response

When working with Python requests, sometimes we need to send multiple HTTP requests and it can take a lot of time to get the response from each request. In such cases, we can use asynchronous requests to speed up the process. With asynchronous requests, we can send multiple requests at once and get the response as soon as it is available for each request.

Using AsyncIO Library

Python provides an AsyncIO library that can be used for asynchronous programming. We can use AsyncIO with requests library to send asynchronous requests. To use AsyncIO with requests library, we need to install the aiohttp library, which is a wrapper around the requests library for AsyncIO.

Here's an example:


      import aiohttp
      import asyncio

      async def get_response(url):
          async with aiohttp.ClientSession() as session:
              async with session.get(url) as response:
                  return await response.text()

      async def main():
          urls = [
              'https://www.google.com',
              'https://www.github.com',
              'https://www.linkedin.com'
          ]
          tasks = []
          for url in urls:
              tasks.append(asyncio.ensure_future(get_response(url)))
          responses = await asyncio.gather(*tasks)
          print(responses)

      loop = asyncio.get_event_loop()
      loop.run_until_complete(main())

In the above example, we define a function 'get_response' that takes a URL as input and returns the response text. We use the aiohttp library to send HTTP requests asynchronously. We then define another function 'main' that creates a list of URLs and creates tasks for each URL using the 'get_response' function. We then use the 'asyncio.gather' function to wait for all the tasks to complete and get the responses. Finally, we print the responses.

Using Concurrent.Futures Library

Python also provides a Concurrent.Futures library that can be used for asynchronous programming. We can use Concurrent.Futures with requests library to send asynchronous requests. To use Concurrent.Futures with requests library, we need to use the ThreadPoolExecutor or ProcessPoolExecutor classes.

Here's an example:


      import concurrent.futures
      import requests

      def get_response(url):
          response = requests.get(url)
          return response.text

      def main():
          urls = [
              'https://www.google.com',
              'https://www.github.com',
              'https://www.linkedin.com'
          ]
          with concurrent.futures.ThreadPoolExecutor() as executor:
              results = executor.map(get_response, urls)
              for result in results:
                  print(result)

      if __name__ == '__main__':
          main()

In the above example, we define a function 'get_response' that takes a URL as input and returns the response text. We use the requests library to send HTTP requests. We then define another function 'main' that creates a list of URLs and uses the ThreadPoolExecutor class to send requests asynchronously. We then use the 'executor.map' function to get the responses and print them.

Both AsyncIO and Concurrent.Futures libraries can be used for asynchronous programming with Python requests library. Which one to use depends on your specific use case and preference.

Share