Retry Flaky AI Requests Without Wasting Your Budget

Build your retry policy around three questions: is another attempt likely to help, is repeating the request safe, and how much more time and money are you willing to spend? Work through the following steps on one small request before applying the policy to your whole project.

First, find out who already retries

Check your SDK and any application framework before adding a retry loop. The official OpenAI Python SDK already retries certain connection failures and HTTP errors. An OpenAI-compatible connection does not mean every client has identical retry behavior, so inspect the client you actually use. ([github.com](https://github.com/openai/openai-python/blob/main/README.md?plain=1&utm_source=openai))

Choose one place to manage retries. For example, let the SDK handle them and keep your application from automatically restarting the entire operation. If both layers allow three attempts, one user action can produce nine downstream attempts. Layered retries multiply rather than simply add together. ([aws.amazon.com](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/))

Next, separate temporary failures from fixable mistakes

Read the error details before deciding what happens next. Connection failures and temporary server errors can justify another attempt. Invalid credentials, malformed requests, and permission problems generally require a correction instead. A 429 response also needs inspection: temporary rate limiting and exhausted credits call for different actions. Waiting cannot replenish a balance. ([developers.openai.com](https://developers.openai.com/api/docs/guides/error-codes?utm_source=openai))

Write your policy as a small set of explicit decisions. Retry recognized temporary failures within your limits. Stop on authentication, request-validation, and billing errors. Treat an unfamiliar error as something to investigate rather than permission to keep sending requests.

Set a deadline before choosing the waiting pattern

Distinguish the time allowed to establish a connection from the time allowed to receive a response. A timeout that is too short can trigger unnecessary retries, while an excessively long one leaves work waiting. Base your settings on observed response times and allow room for internet latency. ([aws.amazon.com](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/))

For a small interactive experiment, consider one initial attempt and at most two retries, all inside a single overall deadline. Treat these as starting limits, not universal defaults. Include both request time and waiting time, and stop when the user cancels.

Space retries out instead of repeating immediately

Use exponential backoff with jitter when the server supplies no waiting instruction. That means increasing the waiting window after each failure and choosing a random delay within it. For your first experiment, try windows of up to one second and then two seconds. Cap both attempts and elapsed time. ([help.openai.com](https://help.openai.com/en/articles/5955604-troubleshooting-api-rate-limits-and-429-errors?utm_source=openai))

When a valid Retry-After instruction is present, wait at least that long. If the requested wait extends beyond your overall deadline, stop and explain that the request can be tried later. Do not shorten the server's delay merely to fit another attempt into your schedule. ([developers.openai.com](https://developers.openai.com/api/docs/guides/error-codes?utm_source=openai))

Handle uncertain completion without promising free retries

A timeout tells you that your client stopped waiting, not that the remote operation never happened. Repeating work can duplicate its effects. Use a supported idempotency mechanism when one exists, but do not assume that adding an arbitrary identifier makes an endpoint safe to repeat. ([aws.amazon.com](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/))

For AI requests, budget conservatively: do not assume an interrupted attempt was free. If a stream breaks after displaying text, preserve that partial answer and offer an explicit restart. Avoid silently stitching a fresh generation onto it or automatically repeating actions that generated tool instructions might already have triggered.

Test the policy and inspect the spending

Start with a short prompt and a modest output allowance. Simulate a connection failure, a temporary server error, and a non-retryable authentication error. Confirm that only the intended failures trigger another attempt and that cancellation prevents additional sends.

Use MyRelay's request-level token and cost records to compare what your application reports with what reached the relay. Keep a local record of attempt counts, delays, and outcomes without logging secrets or sensitive prompts. Create a MyRelay account to run this small test with one key and a budget you choose.

Finish by checking total cost per completed task, not just the number of successful responses. Keep the retry limits conservative until your own measurements justify changing them.