Menu
Article Troubleshooting

retry-after python requests

Narkw Brahma

Narkw Brahma

• 2 min read

Retrying After Python Requests

If you are working with Python requests library to send HTTP requests, you might encounter situations where the request fails due to various reasons like network congestion, server-side issues, etc. In such cases, you might want to retry the request after a certain amount of time to avoid overloading the server with too many failed requests.

Fortunately, the requests library provides a simple way to implement retries using the Retry object from the urllib3.util.retry module.

Using Retry Object

To use the Retry object, you need to create an instance of it and pass it as a parameter to the Session object. The Retry object takes several parameters like total, backoff_factor, etc. You can customize these parameters based on your requirements.


import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

retry_strategy = Retry(
    total=3, # Number of retries
    backoff_factor=2 # Delay between retries in seconds (exponential backoff)
)

adapter = HTTPAdapter(max_retries=retry_strategy)

# Create a session and mount the adapter
session = requests.Session()
session.mount('http://', adapter)
session.mount('https://', adapter)

url = 'https://example.com'
response = session.get(url)

The above code creates a Retry object with a maximum of 3 retries and a delay of 2 seconds between retries (exponential backoff). It then creates an HTTP adapter with this retry strategy and mounts it on a session object for all requests.

You can also customize the Retry object further by adding specific status codes to retry for and excluding certain status codes from retries. For example, to retry only for 500 and 502 status codes, you can modify the Retry object as follows:


retry_strategy = Retry(
    total=3, # Number of retries
    backoff_factor=2, # Delay between retries in seconds (exponential backoff)
    status_forcelist=[500, 502] # Status codes to retry for
)

Alternatively, you can exclude certain status codes from retries by using the status_exclude parameter:


retry_strategy = Retry(
    total=3, # Number of retries
    backoff_factor=2, # Delay between retries in seconds (exponential backoff)
    status_exclude=[404] # Status codes to exclude from retries
)

Using Retry-After Header

Sometimes, the server might return a Retry-After header in the response indicating how long you should wait before retrying the request. In such cases, you can extract the value of the header and use it to delay the next retry using Python's time.sleep() function.


import requests
import time

url = 'https://example.com'
response = requests.get(url)

if response.status_code == 429:
    # Too many requests, wait for Retry-After header value
    retry_after = int(response.headers.get('Retry-After', '5'))
    time.sleep(retry_after)
    response = requests.get(url)

The above code checks if the response status code is 429 (Too Many Requests) and extracts the value of the Retry-After header. It then waits for that many seconds before retrying the request.

Conclusion

Retrying failed requests is an important aspect of building robust HTTP clients. The requests library provides a simple way to implement retries using the Retry object and also allows you to handle Retry-After headers from the server. With these techniques, you can build HTTP clients that gracefully handle temporary failures and avoid overloading the server with too many failed requests.

Share