When building AI-powered features, the most frustrating part isn't the model itself—it's the API layer that wraps it. I've spent years debugging production issues where a single misconfigured timeout or retry policy caused cascading failures across dozens of services. The root problem? AI endpoints are inherently unstable: they can return partial results, fail silently, or take unpredictable amounts of time.
The solution is to treat these endpoints like any other API, but with stricter boundaries. This post focuses on how to design API boundaries that handle the unique challenges of AI model calls through timeouts, retries, and idempotency patterns. These patterns aren't just best practices—they're survival tactics for maintaining reliability in AI systems.
The Hidden Cost of Unbounded AI Calls
Let's start with a simple example: a chatbot endpoint that calls an LLM. If the model returns a partial response, the API might hang indefinitely, causing the client to time out. Worse, if the API doesn't handle this properly, the user might see a broken response or get stuck in an infinite loop.
The key insight is that AI models are not deterministic. They can fail silently, return incomplete results, or have variable latency. This means we need to treat every model call like a potential race condition.
A common mistake is to rely on the model's own timeout mechanisms. For example, if you call an OpenAI endpoint and set timeout: 60s in your client, you're still exposing your system to the model's own latency spikes. Instead, you should define timeouts at the API layer, not the model layer. This gives you more control over the system's behavior.
// Example: Enforce 3s timeout at the API layer, not the model
const response = await fetch('https://api.example.com/generate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prompt: 'Hello' }),
timeout: 3000, // 3 seconds
});This approach ensures that your system doesn't wait for the model to finish its processing. It also makes it easier to implement retries and fallbacks without introducing complexity into the model call itself.
Retry Policies That Don't Break the System
Retry logic is a double-edged sword. Used properly, it can help your system recover from transient failures. Used poorly, it can turn a single error into a cascading outage. The key is to implement exponential backoff with a maximum retry limit, and to avoid retrying every possible error.
For example, if an AI endpoint returns a 503 error, it's worth retrying. But if it returns a 400 error (like invalid input), there's no point in retrying. The same applies to 429 errors (rate limiting) or 500 errors (server-side issues).
Here's a simple retry policy I've used in production:
- First retry: 500ms delay
- Second retry: 1s delay
- Third retry: 2s delay
- Maximum retries: 3
This gives the system a chance to recover from transient failures without overwhelming the backend. It also prevents the API from getting stuck in an infinite retry loop.
// Example: Simple retry logic with exponential backoff
const maxRetries = 3;
let retryCount = 0;
let delay = 500;
while (retryCount < maxRetries) {
try {
const response = await fetch('https://api.example.com/generate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prompt: 'Hello' }),
});
if (response.ok) {
return await response.json();
}
throw new Error(`Request failed with status ${response.status}`);
} catch (error) {
retryCount++;
if (retryCount >= maxRetries) {
throw new Error('Max retries exceeded');
}
await new Promise(resolve => setTimeout(resolve, delay));
delay *= 2; // Exponential backoff
}
}This approach ensures that your system doesn't get stuck in a retry loop while also giving the backend a chance to recover from transient issues. It's important to note that this logic should be implemented at the API layer, not the model layer.
Idempotency Tokens for Consistent State
One of the most underappreciated patterns in API design is idempotency. For AI endpoints, this means ensuring that repeated requests for the same operation don't cause unintended side effects. For example, if a user submits the same prompt twice, the system should handle both requests in a way that doesn't create duplicate results or corrupted state.
The solution is to use idempotency tokens. These are unique identifiers that the client generates for each request. The server then uses this token to track whether the request has already been processed.
Here's how it works in practice:
- The client generates a unique idempotency token for each request.
- The client sends the token along with the request.
- The server checks if the token has been used before.
- If the token is new, the server processes the request.
- If the token is already in use, the server returns the cached result.
This approach ensures that the same request doesn't get processed multiple times, which is critical for maintaining consistency in AI systems. It also allows the client to retry requests without worrying about duplicate results.
// Example: Using idempotency tokens in an AI endpoint
const idempotencyToken = 'abc123';
const response = await fetch('https://api.example.com/generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Idempotency-Token': idempotencyToken
},
body: JSON.stringify({ prompt: 'Hello' }),
});This pattern is especially useful for operations that can take a long time to complete, like generating a long text response. It ensures that the client can safely retry the request without causing the server to process the same request multiple times.
Practical Tradeoffs in Production
In practice, these patterns require careful tradeoffs. For example, setting a timeout that's too short can cause legitimate requests to fail. Setting it too long can allow the system to hang indefinitely. The same applies to retry policies and idempotency tokens.
One common mistake is to treat AI endpoints like traditional APIs. For example, using the same timeout and retry policies as a database query. This doesn't work because AI models have different latency characteristics. You need to adjust your timeouts and retries based on the model's behavior.
Another common mistake is to skip idempotency tokens for certain operations. This can lead to duplicate results or corrupted state, especially when dealing with streaming responses. It's important to use idempotency tokens for all operations that can take a long time to complete.
Conclusion
Designing stable AI endpoints requires a combination of timeouts, retries, and idempotency patterns. These patterns aren't just best practices—they're survival tactics for maintaining reliability in AI systems. By treating AI model calls like any other API, you can build a system that's resilient to failures and maintains consistency under pressure.
The key takeaway is to always define timeouts and retries at the API layer, not the model layer. And never skip idempotency tokens for operations that can take a long time to complete. These patterns will help you build a system that's both reliable and maintainable.