Menu
Article Bugs

python requests retry best practices

Narkw Brahma

Narkw Brahma

• 2 min read

Python Requests Retry Best Practices

As someone who has worked with Python requests frequently, I have come across situations where the request fails due to various reasons like network issues, server overload, or simply due to temporary server unavailability. In such cases, it is important to retry the request to ensure successful completion of the task at hand

1. Retrying with simple loop:

One of the easiest and quick ways to retry a failed request is to use a simple loop. This can be done by simply wrapping the request code in a loop and adding some logic to handle the retry count and delay between retries.

import requests
import time

url = 'https://example.com'

retry_count = 3
retry_delay = 2

for i in range(retry_count):
    try:
        response = requests.get(url)
        response.raise_for_status()
        break
    except requests.exceptions.RequestException as e:
        print(f'Retrying {i+1}/{retry_count}...')
        time.sleep(retry_delay)
else:
    raise Exception('Failed to fetch data after multiple retries')

In the code above, we are retrying the request for a maximum of 3 times with a delay of 2 seconds between each retry. If the request fails after all the retries, it raises an exception indicating that the request failed even after multiple attempts.

2. Using Retry Library:

Another way to handle retries in Python Requests is to use the retry library. This library provides a simple and powerful way to handle retries with some advanced features like exponential backoff and jitter.

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

url = 'https://example.com'

retry_count = 3
retry_backoff_factor = 2

session = requests.Session()

retry = Retry(
    total=retry_count,
    backoff_factor=retry_backoff_factor,
    status_forcelist=[500, 502, 503, 504]
)

adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

response = session.get(url)
response.raise_for_status()

The code above uses the Retry library to handle retries. We define a retry object with a maximum of 3 retries and a backoff factor of 2. We also define the list of status codes for which we want to retry the request. Finally, we create a session and mount the HTTPAdapter with the retry object. This will ensure that any request made using this session will be retried automatically based on the defined retry object.

Conclusion:

Handling retries is an important aspect of making robust and resilient requests. Whether it is a simple loop or a powerful library like Retry, handling retries can make a big difference in the success of your requests. Choose the method that suits your needs and ensure that your code is reliable and scalable.

Share