Web Scraping at Scale: How to Collect Millions of Pages Reliably

A collector that handles a few thousand pages is a fetch loop with some error handling bolted on. The same code aimed at millions of pages becomes a different kind of system, because at that volume the exceptions stop being exceptions and become the workload. Web scraping at scale is not the small problem multiplied; it is a reliability problem that arrives disguised as a throughput problem.

What makes it expensive is how quietly it fails. A job ends, the exit code is zero, the request-rate graph looks healthy, and the dataset that lands is missing eight percent of the catalog. Nobody reading it can distinguish a category that was genuinely empty from a category that was never collected successfully.

This article works outward from that failure: what changes when volume arrives, the architecture that absorbs it, and how to operate a pipeline for usable yield rather than raw rate. At smaller volumes, the choice of collection method still matters. Our guide to modern web scraping covers the main strategies for building reliable data collection before scale becomes the dominant challenge.

What changes when collection leaves small scale

At small volume, a run either works or throws: you see the exception, fix it, and re-run. At millions of pages, no run is fully successful – some slice of it never is – and the useful question changes from “did it fail?” to “what fraction failed, which fraction, and does anyone know?”

Four things behave differently once you cross that line.

Partial completion becomes the normal outcome. A run that covers 92% of its targets is not a broken run; it is an ordinary one. That is fine when the number is measured and inside an agreed tolerance, and dangerous when the only signal is that the process ended, because the gap between 100% and 92% is invisible to everyone reading the output.

Retries amplify load instead of absorbing it. A retry policy that is harmless at a thousand requests is a load generator at a million. When a target slows down, every in-flight request against it fails at roughly the same moment, and a fleet-wide retry adds traffic exactly when the source has least capacity to serve it.

Bottlenecks become shared. At small scale, one slow target costs one slow job. At high volume, the worker pool, the queue, and the write path are shared by the whole workload, so a single pathological source – a listing site now answering in eight seconds instead of 300 milliseconds – consumes capacity hundreds of well-behaved targets were relying on. Nothing crashes. Everything gets slower.

Targets push back. At volume you will meet rate limits and the occasional block, and those are a property of the environment rather than an incident to resolve. Treat them as a capacity input.

None of this is a coding defect; it is what happens when a fetch script is asked to behave like infrastructure.

Architecture that can take the volume

The architecture for web scraping at scale is the one where each of those failures has somewhere to go.

Three decisions do the real work.

Isolation is what concurrency buys you. More workers is a capacity change; isolation is a reliability change. Because a worker’s only contract is to take a unit of work and return a validated result, one worker dying mid-fetch costs a unit rather than a run.

Sharding keeps a hot target from stalling the fleet. In one undifferentiated queue, every unit of work competes for every worker, so a source that is slow, heavy, or rate-limited occupies workers in proportion to how badly it behaves. Partitioning the queue – by target, by workload class, by the kind of fetch required – caps the damage any one source can do. This kind of workload isolation can also be implemented at the queue level; our deep dive into BullMQ-Pro’s Group feature explores one practical approach to isolating workflows within shared queue infrastructure. The heavy catalog gets its own lane and its own workers; everything else proceeds at its own pace. That is the difference between scalable web scraping and a fast script that occasionally seizes.

Backpressure decides what happens when a hop can’t keep up. Something is always the slowest part of the path, usually the write side. Without backpressure, intake keeps producing at its natural rate and the queue absorbs the imbalance until it can’t. With it, a slow sink slows the admission of new work, so the system degrades visibly instead of accruing a backlog you discover at capacity.

Retries, failures, and fault tolerance

Fault tolerance at volume starts with classification. A generic “retry three times” treats every failure as the same failure, and they are not.

• Retryable. Timeouts, connection resets, transient server errors. The request was reasonable and the answer was unlucky, so trying again with exponential backoff and jitter is likely to work.
• Poison. A URL that no longer exists, a page the parser cannot handle, a payload that fails validation. Retrying is guaranteed to fail again and burns capacity doing it. Quarantine these, count them, get them in front of a human.
• Policy. The source is telling you to slow down. That is not an error to retry against; it is a signal to lower the rate for that target and reschedule the work.

Collapsing the three into one path is how retry storms start. Unbounded retries against a struggling source turn partial degradation into a self-inflicted load spike, so retry budgets belong per target and per unit of work, with a circuit breaker that stops hammering a source that is plainly unavailable. A dead-letter path for the work you give up on is how the failure stays countable.

Because retries mean the same page will sometimes be fetched twice, writes have to be idempotent. Give each unit of work a stable identity and make the merge step converge on the same state whether it is applied once or three times.

Gaps and duplicates are both delivery bugs, and to the consumer they are the same bug: the data does not describe what the source contained, and nothing in the output admits it. A single missed fetch is not evidence that a listing was removed, so never let one absence delete a record.

Rate limits and load

Source policy is a capacity constraint, not an obstacle. Every target has a rate it will serve without degrading, and that rate – not your worker count – is the ceiling on how fast that slice of the workload can be collected. When reliable access itself becomes part of the scaling challenge, a managed web data unblocker can separate access infrastructure from the rest of the collection pipeline. Accept that, and load planning becomes an allocation problem.

Give each target its own budget – concurrent requests, requests per interval, and a rule for what happens when it starts refusing – derived from observed behaviour rather than a fixed guess. Then enforce fairness, because the natural failure mode of a shared queue is that whichever target has the most pending work takes most of the capacity. Weighted scheduling and per-target caps keep the long tail from starving behind one enormous job.

Queues are the shock absorber that makes this survivable: load arrives in bursts, sources vary in speed hour to hour, and a queue lets intake stay steady while consumption moves underneath it. What a queue cannot do is create capacity. If arrival exceeds what your budgets allow for long enough, the queue is not smoothing anything – it is where the backlog lives.

Throughput is not success

Requests per second measures how hard the system is working, not how much it delivered. The number that matters is usable yield: success rate multiplied by completeness. A pipeline at 2,000 requests per second with an 80% success rate against 90% of its targets delivers less than a slower one that gets nearly all of both, and costs more to run.

The stall points are the hops themselves: the queue, when arrival outruns consumption; the worker pool, when capacity or parsing cost is the limit; the sink, when the write path cannot accept results as fast as they are produced; and the source, when its own limits bind first. Only one of those is fixed by adding workers, which is why the reflex to scale the fetch layer so often changes nothing.

Observability – knowing when the pipeline is lying

A collection pipeline at volume is very good at looking healthy while delivering badly. Four measurements close most of that gap.

Queue age, not just depth. Depth says how much work is outstanding, not whether any of it is moving. The age of the oldest pending item is the real lag, and it is what exposes a partition being quietly starved while total depth looks stable.

Success rate per target, not in aggregate. A fleet-wide 95% is a comfortable number that can contain one source sitting at 20%. Aggregates hide precisely the failures that produce incomplete datasets, so break the rate down by target and workload class.

Retry ratio. Retries as a share of total requests is an early warning. When it climbs while the success rate holds, the system is spending more capacity for the same result – the shape that precedes a retry storm.

Sink lag. The last hop is the one nobody instruments. Measure the time from a successful fetch to the result being readable by a consumer; a fast collection layer with a slow write path is a slow system, whatever the fetch metrics claim.

Then alert on yield rather than process uptime. Every worker being healthy is a statement about your infrastructure, not your data. The alerts worth keeping fire when completeness drops below tolerance, when queue age passes the point where the output stops being useful, or when a target’s success rate collapses.

Takeaways

• Scale is a reliability problem first. Request rate is not the product; a complete dataset is.
• Queues buy isolation and backpressure – and hide lag if you only watch depth.
• Classify failures into retryable, poison, and policy. Unbounded retries are a load amplifier, not a safety net.
• Measure usable yield – success rate and completeness – not vanity throughput.
• Observe queue age, per-target success, and sink lag, and put your alerts there.
• A fast fetch layer with a slow or lossy sink is not a reliable collection system. The slowest hop defines what you deliver.