API latency: benchmarks, causes and fixes

Every 100ms of API latency costs you users. Google's research showed a 0.5-second slowdown reduced search traffic by 20%. Amazon found each 100ms delay reduced revenue by 1%. These are extreme cases, but the underlying dynamic applies to any API — latency compounds into abandonment.

This guide covers what good latency actually looks like at each percentile, the main causes of high latency in production APIs, and how to allocate a latency budget across the components of a request.

What good latency looks like: real benchmarks

These are p50, p95, and p99 latency numbers for common API configurations, measured from the server side (not including network time to the client). Workload: authenticated JSON API returning a single resource after a database read.

The key insight from these numbers: there's very little practical difference between FastAPI, Express and Fastify at p50 — they're all fast. The difference shows at p99, where runtime overhead and garbage collection start to matter. If your p99 is 3x your p50, you have a tail latency problem that's likely GC or connection pool exhaustion.

The four main causes of high API latency

1. N+1 database queries

The classic mistake: your endpoint fetches a list of 50 items, then makes a separate database query for each item to get related data. That's 51 queries instead of 2. Each query takes 2–5ms, and suddenly your endpoint takes 150–250ms instead of 10ms.

# Bad: N+1 queries
users = db.query("SELECT * FROM users LIMIT 50")
for user in users:
    user.orders = db.query(f"SELECT * FROM orders WHERE user_id = {user.id}")

# Good: single JOIN query
users = db.query("""
  SELECT u.*, o.id as order_id, o.total, o.status
  FROM users u
  LEFT JOIN orders o ON o.user_id = u.id
  LIMIT 50
""")

2. Missing connection pooling

Opening a new database connection for every request adds 20–100ms of overhead. A connection pool keeps connections alive and reuses them. PgBouncer for PostgreSQL, or the built-in pooling in most ORMs, should always be enabled in production.

3. Synchronous external API calls

If your endpoint calls an external API (Stripe, SendGrid, a third-party data service) synchronously, your latency is your latency plus their latency. If they're having a slow day (p99 latency of 800ms), your endpoint inherits that. Move external calls to background jobs when possible, or parallelize them when you need the results.

4. No HTTP caching

API responses that don't change frequently should be cached at the HTTP level. A properly set Cache-Control header means repeat requests for the same data never hit your server at all.

# For responses that change rarely (product catalog, config)
response.headers['Cache-Control'] = 'public, max-age=300, stale-while-revalidate=60'

# For user-specific responses
response.headers['Cache-Control'] = 'private, max-age=60'

# For truly dynamic responses that must be fresh
response.headers['Cache-Control'] = 'no-store'

Latency budget calculator

A latency budget breaks your total acceptable latency into allocations for each component of a request. If your target is 200ms end-to-end, you need to plan where each millisecond goes.

The 80/20 of latency optimization

If you have to pick two things to fix first, pick these:

1. Find and fix N+1 queries — use your ORM's query logging in development to see every query that fires on a single request. If you see more than 3–4 queries for a simple read endpoint, you have an N+1 problem.
2. Add response caching for stable data — even a 5-minute cache on product data or configuration endpoints eliminates database load and reduces latency to near zero for cached responses.

Everything else — connection pooling, async external calls, choosing a faster runtime — has a real impact but is secondary to these two. Get N+1 and caching right first.