Building a Scalable URL Monitoring System

· 5 min read · Live Project

When we started building HTTP Tiger, the team thought it would be straightforward. Three months later, we had learned hard lessons about rate limiting, session management, and building reliable monitoring at scale.

The Rate Limiting Problem

Users wanted to check 100+ URLs at once, but naive concurrent requests would get our IP banned by target servers.

Domain-based throttling with configurable concurrency limits and exponential backoff.

// Group URLs by domain, then process with delays
const chunks = chunkByDomain(urls);

for (let i = 0; i < chunks.length; i += maxConcurrent) {
    const batch = chunks.slice(i, i + maxConcurrent);
    await Promise.allSettled(
        batch.map(chunk => processChunk(chunk))
    );
    await delay(rateLimitMs);
}

The Silent Failure Bug

Users reported URLs showing as "failed" when they were clearly working. The culprit: CORS policies and mixed content blocking in the browser. We solved it with a hybrid approach - try direct fetch first, fall back to server-side proxy.

// Client-side attempt, then server-side fallback
try {
    await fetch(url, { mode: 'no-cors', method: 'HEAD' });
    return { status: 'reachable', method: 'direct' };
} catch {
    const proxy = await fetch('/api/check-url', {
        method: 'POST',
        body: JSON.stringify({ url })
    });
    return await proxy.json();
}

Lessons Learned

  1. Design rate limiting first - not as an afterthought
  2. Always have a fallback when external services fail
  3. Show progress - users need to know something is happening
  4. Monitor real usage - edge cases you never imagined will appear