Building a Scalable URL Monitoring System

Written by Terrell Flautt on October 7, 2025

· 12 min read · Live Project

When I set out to build URL Status Checker, I thought it would be a simple weekend project. "Just ping some URLs and show the status codes," I told myself. Three months later, I had learned painful lessons about rate limiting, user session management, and the complexity of building a reliable monitoring service.

The Initial Challenge

🚨 Problem

Users wanted to check 100+ URLs at once, but naive implementation would either crash the browser or get IP-banned by target servers.

The first version was embarrassingly simple:

// Don't do this - it will break everything
async function checkUrls(urls) {
    const results = [];
    for (const url of urls) {
        const response = await fetch(url);
        results.push({ url, status: response.status });
    }
    return results;
}

This approach had several critical flaws:

Solution 1: Intelligent Rate Limiting

✅ Solution

Implemented a sophisticated rate limiting system with exponential backoff and domain-based throttling.

class URLChecker {
    constructor() {
        this.domainQueues = new Map();
        this.maxConcurrent = 5;
        this.domainDelay = 1000; // 1 second between requests to same domain
    }

    async checkUrlsBatch(urls, onProgress) {
        const chunks = this.chunkByDomain(urls);
        const results = [];

        for (let i = 0; i < chunks.length; i += this.maxConcurrent) {
            const batch = chunks.slice(i, i + this.maxConcurrent);
            const batchResults = await Promise.allSettled(
                batch.map(chunk => this.processChunk(chunk))
            );

            results.push(...batchResults.flat());
            onProgress(results.length, urls.length);

            // Rate limit between batches
            if (i + this.maxConcurrent < chunks.length) {
                await this.delay(500);
            }
        }

        return results;
    }

    chunkByDomain(urls) {
        const domainChunks = new Map();

        urls.forEach(url => {
            const domain = new URL(url).hostname;
            if (!domainChunks.has(domain)) {
                domainChunks.set(domain, []);
            }
            domainChunks.get(domain).push(url);
        });

        return Array.from(domainChunks.values());
    }
}

The Authentication Complexity

Initially, I thought I'd just store favorites in localStorage. Then users started asking: "Can I access my lists from multiple devices?" This led me down the rabbit hole of user authentication and session management.

🚨 Problem

Users needed persistent storage across devices, but I wanted to avoid the complexity of user accounts and passwords.

Google Sign-In seemed like the perfect solution, but integrating it properly was trickier than expected:

class AuthManager {
    constructor() {
        this.user = null;
        this.isInitialized = false;
    }

    async initialize() {
        return new Promise((resolve) => {
            gapi.load('auth2', async () => {
                try {
                    this.auth2 = await gapi.auth2.init({
                        client_id: '463808979755-your-client-id.googleusercontent.com'
                    });

                    // Check if user is already signed in
                    if (this.auth2.isSignedIn.get()) {
                        this.user = this.auth2.currentUser.get();
                        this.updateUI();
                    }

                    this.isInitialized = true;
                    resolve();
                } catch (error) {
                    console.error('Auth initialization failed:', error);
                    this.gracefulFallback();
                    resolve();
                }
            });
        });
    }

    gracefulFallback() {
        // If Google Auth fails, fall back to localStorage
        console.log('Using localStorage fallback for user data');
        this.updateUI();
    }
}

The Pricing Model Dilemma

This was where business meets technical implementation. I needed to create a system that was:

✅ Solution

Implemented a credit-based system with different rate limits per plan tier.

const PLANS = {
    free: {
        name: 'Free',
        monthlyChecks: 500,
        batchSize: 10,
        rateLimit: 5000, // 5 seconds between batches
        features: ['Basic checking', 'Export results']
    },
    pro: {
        name: 'Pro',
        monthlyChecks: 10000,
        batchSize: 50,
        rateLimit: 2000, // 2 seconds between batches
        features: ['Priority checking', 'Favorites', 'API access']
    },
    business: {
        name: 'Business',
        monthlyChecks: 100000,
        batchSize: 100,
        rateLimit: 1000, // 1 second between batches
        features: ['Unlimited favorites', 'Team access', 'Webhooks']
    }
};

class PlanManager {
    checkUserLimits(user, requestedChecks) {
        const plan = PLANS[user.plan || 'free'];
        const usage = this.getMonthlyUsage(user);

        if (usage + requestedChecks > plan.monthlyChecks) {
            throw new Error(`Monthly limit exceeded. Upgrade to check more URLs.`);
        }

        if (requestedChecks > plan.batchSize) {
            throw new Error(`Batch size too large. Maximum ${plan.batchSize} URLs per check.`);
        }

        return {
            allowed: true,
            remaining: plan.monthlyChecks - usage,
            rateLimit: plan.rateLimit
        };
    }
}

Performance Optimization Discoveries

The biggest performance wins came from unexpected places:

1. Debounced Input Validation

Users would paste large lists of URLs, and validating each one on every keystroke was killing performance.

const debouncedValidation = debounce((urls) => {
    const results = urls.map(url => ({
        url,
        isValid: this.isValidUrl(url),
        issues: this.validateUrl(url)
    }));
    this.updateValidationUI(results);
}, 300);

2. Progressive Results Loading

Instead of waiting for all URLs to complete, I streamed results as they came in.

async processUrlStream(urls, onResult) {
    const stream = new ReadableStream({
        async start(controller) {
            for (const url of urls) {
                try {
                    const result = await checkSingleUrl(url);
                    controller.enqueue(result);
                    onResult(result); // Update UI immediately
                } catch (error) {
                    controller.enqueue({ url, error: error.message });
                }
            }
            controller.close();
        }
    });

    return stream;
}

The Hardest Bug: Silent Failures

The most frustrating bug took weeks to track down. Users reported that some URLs would show as "failed" when they were clearly working. The issue? CORS policies and mixed content blocking.

🚨 Problem

Browser security policies prevented direct URL checking for many sites, but errors were being silently swallowed.

The solution required a hybrid approach:

async function checkUrlWithFallback(url) {
    try {
        // Try direct fetch first
        const response = await fetch(url, {
            mode: 'no-cors',
            method: 'HEAD'
        });
        return { url, status: 'reachable', method: 'direct' };
    } catch (directError) {
        try {
            // Fallback to server-side check
            const proxyResponse = await fetch('/api/check-url', {
                method: 'POST',
                headers: { 'Content-Type': 'application/json' },
                body: JSON.stringify({ url })
            });
            const result = await proxyResponse.json();
            return { url, ...result, method: 'proxy' };
        } catch (proxyError) {
            return {
                url,
                status: 'failed',
                error: 'Unable to reach URL',
                method: 'none'
            };
        }
    }
}

Lessons Learned

  1. Start with constraints: Rate limiting should be designed first, not added later.
  2. Graceful degradation: Always have a fallback when external services fail.
  3. User feedback: Show progress, explain delays, and communicate limitations clearly.
  4. Plan for scale: Even simple tools can get popular quickly.
  5. Monitor real usage: Users will find edge cases you never considered.

Current Status & Future Plans

URL Status Checker now serves thousands of checks daily with a 99.9% uptime. The next major features in development:

Building a "simple" URL checker taught me that there's no such thing as a simple web service. Every feature decision creates ripple effects, and the real challenges only emerge when users start pushing your assumptions.