Most real scraping jobs are not "grab one page." They are "collect every product in a category," "every job in a board," "every listing in a directory." That is list crawling: systematically walking a set of listing pages to gather items, then optionally following each to its detail page. Master this pattern and you can scrape almost any structured site. This guide shows the two-phase approach in Python, how to handle every pagination style, and how to run it at scale.
What Is List Crawling?
List crawling is different from scraping a single page. A listing page (a category, search result, or index) shows many items in a repeating structure, usually across multiple pages. List crawling means:
- Crawl the listing pages — follow pagination to collect the link to every item.
- Visit each item — fetch each detail page and extract its full data.
Splitting the job into these two phases keeps it reliable: you get a complete, deduplicated list of URLs first, then process them independently. It is the same idea behind frameworks like Scrapy, and it pairs naturally with the fundamentals in building a web scraper in Python. For the distinction between crawling and scraping, see web crawling vs web scraping.
Phase 1: Crawl the Listing Pages
Walk through the paginated list and collect every item link. Using the practice site books.toscrape.com as a safe example, and routing through a residential proxy:
import requests
from urllib.parse import urljoin
from bs4 import BeautifulSoup
proxy = "http://USERNAME:[email protected]:12321"
proxies = {"http": proxy, "https": proxy}
BASE = "https://books.toscrape.com/catalogue/"
def collect_links(start="page-1.html"):
links, page = [], start
while page:
soup = BeautifulSoup(requests.get(BASE + page, proxies=proxies, timeout=20).text, "html.parser")
for a in soup.select("article.product_pod h3 a"):
links.append(urljoin(BASE, a["href"]))
nxt = soup.select_one("li.next a") # follow the "next" link
page = nxt["href"] if nxt else None
return linksNotice the loop follows the "next" link until there is not one — a robust way to reach the last page without guessing how many pages exist.
Phase 2: Visit Each Item
With the full link list in hand, fetch each detail page and pull its fields:
def scrape_item(url):
soup = BeautifulSoup(requests.get(url, proxies=proxies, timeout=20).text, "html.parser")
return {
"url": url,
"title": soup.select_one("h1").get_text(strip=True),
"price": soup.select_one(".price_color").get_text(strip=True),
"availability": soup.select_one(".availability").get_text(strip=True),
}
links = collect_links()
data = [scrape_item(u) for u in links]
print(len(data), "items")Handling Every Pagination Style
Real sites paginate in different ways — identify which one you are dealing with:
- Next link: follow the "next" button until it disappears (as above). The most robust.
- Numbered pages: increment a
page=oroffset=parameter until a page returns no items. - Load more / infinite scroll: the list is loaded by JavaScript. Either call the underlying JSON API the page uses (check the network tab), or render it with Playwright.
- Hidden API: many listing pages fetch items from a JSON endpoint — scraping that directly is faster and cleaner than parsing HTML.
Running It at Scale
Crawling thousands of pages against one site quickly triggers rate limits and bans. Three things keep a large crawl alive:
- Rotate IPs. Route requests through residential proxies so each exits from a different real IP — the difference between finishing a crawl and getting your one IP banned on page 50.
- Add concurrency carefully. Fetch detail pages in parallel with async or a thread pool to cut wall-clock time — see asyncio and concurrency vs parallelism — but cap it so you stay polite.
- Deduplicate and retry. Store seen URLs in a set, retry failed pages with backoff, and cache what you have fetched so a re-run does not repeat work.
Save the collected items to JSON to preserve structure, or flatten to CSV for analysis (see JSON vs CSV).
Common Pitfalls
- Missing the last page by hard-coding a page count — follow the "next" link instead.
- Duplicate URLs when items appear on multiple pages — dedupe before phase two.
- Empty results on JS-rendered lists — the items are not in the initial HTML; use the underlying API or a headless browser.
- Getting banned mid-crawl — almost always a single IP making too many requests; rotate residential proxies.
Frequently Asked Questions
What is list crawling?
List crawling is the pattern of walking through a site's paginated listing pages (a category, search, or index) to collect the link to every item, then visiting each item's detail page to extract its data. It is the two-phase backbone of most large scraping projects.
How do I handle pagination in Python?
The most robust method is to follow the "next" link on each listing page until there is not one. Alternatively, increment a page or offset parameter until a page returns no items. For JavaScript "load more" or infinite scroll, call the underlying JSON API the page uses, or render it with a headless browser like Playwright.
Do I need proxies for list crawling?
For any sizeable crawl, yes. Requesting hundreds or thousands of pages from one site quickly triggers rate limits and IP bans. Rotating residential proxies send each request from a different real IP, so the crawl can finish without a single address getting blocked.
How do I speed up a large crawl?
Fetch detail pages concurrently with async or a thread pool, cap the concurrency so you stay polite, deduplicate URLs, retry failures with backoff, and cache fetched pages so a re-run does not repeat work. Rotating proxies let you raise concurrency without tripping rate limits.
Conclusion
List crawling is the pattern that turns a one-page scraper into a tool that can harvest an entire catalog: collect every item link by following pagination, then visit each detail page. Identify the pagination style, split the job into two phases, and route it all through rotating residential proxies so a big crawl actually finishes.
Crawl at scale without bans: SpyderProxy residential proxies from $2.75/GB — rotating, ethically sourced, across 195+ countries.