Menu
Article Security Check

python requests module retry

Narkw Brahma

Narkw Brahma

• 2 min read

Python Requests Module Retry

If you are working with Python and making HTTP requests, you may come across situations where you need to retry the request if it fails. This is where the Python Requests module's retry function comes in handy. In this article, we will discuss how to use the Python Requests module's retry function.

Why Retry?

Retry is important in situations where the network connection is unreliable or the server is unavailable. In such situations, sending the same request multiple times can increase the chances of success.

Using the Retry Function

The retry function is part of the requests.adapters module. To use it, you need to create an instance of the Retry class and pass it to the HTTPAdapter class.


import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

retry_strategy = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "OPTIONS"]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)

response = http.get("https://jsonplaceholder.typicode.com/todos/1")
print(response.status_code)

In the above code, we are creating an instance of the Retry class with a total of 3 retries and a backoff factor of 1 second. We also have a list of HTTP status codes that will trigger a retry and a list of HTTP methods that can be retried. Next, we create an instance of the HTTPAdapter class and pass the retry strategy to it. Finally, we create a session and mount the HTTPAdapter to it. We then make a request to the JSON Placeholder API and print the response status code.

Customizing the Retry Function

You can further customize the retry function by changing the values of the Retry class parameters. For example, you can increase the total number of retries or change the backoff factor to increase or decrease the time between retries.

You can also customize the status code list and method list to suit your needs. For example, you can add more status codes to be retried or remove some HTTP methods from being retried.

Another way to customize the retry function is to handle exceptions. By default, the retry function will only retry on HTTP status codes that are in the status_forcelist parameter. However, there may be other exceptions that you want to retry on. You can do this by adding an exception handler to the Retry instance.


def handle_exception(e, *args, **kwargs):
    if isinstance(e, requests.exceptions.Timeout):
        return True
    return False

retry_strategy = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "OPTIONS"],
    raise_on_status=False
)
retry_strategy.retry_on_exception = handle_exception

In the above code, we define a custom function handle_exception that checks if the exception is a timeout exception. We then add this function to the Retry instance's retry_on_exception parameter. We also set raise_on_status to False, which means that exception will not be raised if retrying fails.

Conclusion

The Python Requests module's retry function is a useful tool for making HTTP requests in unreliable network conditions. By using the Retry class, you can customize the retry behavior to suit your needs.

Share