The Hardest Part of URL Monitoring Isn't the Monitoring

· 4 min read · httptiger.com

Checking whether a URL is up sounds trivial. Send a request, see if you get a response. When we started building HTTP Tiger, we thought the monitoring itself would be the hard part. It was not. The hard parts were everything surrounding it.

You Will Get Blocked

The first version of HTTP Tiger checked all submitted URLs as fast as possible. Users loved it -- until they submitted 200 URLs, 50 of which pointed to the same domain. Hammering a single server with 50 concurrent HEAD requests is indistinguishable from a denial-of-service attack. Several hosting providers started blocking our IP addresses within the first week.

The fix was domain-aware batching. Before checking anything, the system groups URLs by domain. It then processes one domain at a time with a deliberate delay between requests to the same server. Different domains can be checked in parallel because they are different servers. This slows down large batch checks from seconds to maybe a minute, but it means the service actually works instead of getting banned everywhere.

We also added exponential backoff: if a server responds slowly or returns a 429 (Too Many Requests), the system automatically increases the delay before trying the next URL on that domain.

DNS Failures Are Not HTTP Failures

Early on, we treated all failures the same: the URL is down. But users quickly pointed out that there is a difference between "the server returned a 500 error" and "the domain does not exist." One means the site is having problems. The other means you probably typed the URL wrong.

HTTP Tiger now distinguishes between DNS resolution failures (the domain cannot be found at all), connection timeouts (the server exists but is not responding), SSL certificate errors (the server responds but the certificate is invalid or expired), and HTTP error codes (the server responds but with an error status). Each failure type gets a different explanation so users know whether to check their spelling, contact the site owner, or just wait.

The Silent Failure Bug

This one took weeks to track down. Users reported that certain URLs showed as "failed" in HTTP Tiger even though they could load them fine in a browser. We checked the URLs ourselves -- they worked. We checked from different networks -- they worked. But HTTP Tiger kept marking them as unreachable.

The culprit was browser security policies. The initial version attempted to check URLs directly from the browser using JavaScript. But browsers enforce CORS (Cross-Origin Resource Sharing) restrictions that prevent web pages from making requests to arbitrary domains. The requests were being silently blocked by the browser, not by the target server.

The solution was a hybrid approach. The system first attempts a browser-side check, and if that fails for any reason, it automatically routes the request through a server-side proxy that is not subject to browser security restrictions. The server makes the same request from the backend, where CORS does not apply, and returns the actual result.

What We Learned

  1. Design rate limiting before anything else. It is not a feature you add later -- it shapes the entire architecture.
  2. Classify failures precisely. "It's down" is not enough information to be useful.
  3. Never trust browser-only network requests for a tool that checks arbitrary URLs.
  4. Show progress on long operations. A spinning wheel with no context makes users think the tool is broken.