Building a Bounded Web Crawler with Python's urllib Module

For developers in Melbourne or Sydney running on the nbn, scraping data off the web doesn't require a heavyweight framework. A few lines from Python's standard library are often enough to put together a usable crawler, particularly when you only need to walk a handful of pages deep.

The urllib module has been bundled with Python since the early days, which means there are no extra installs to manage. This walkthrough focuses on building a crawler that respects a maximum depth. That single constraint is what stops a script from accidentally hammering a site into the ground or running for hours on a Friday arvo while you're out grabbing a flat white. We'll use urllib.request for fetching, html.parser for extracting links, and a small deque-based queue to keep track of where we've been.

Fetching Pages with urllib.request

The entry point for any crawler is urllib.request.urlopen. Calling it with a URL returns an HTTPResponse object that behaves like a file handle. Read the body with .read(), decode it as UTF-8 in most cases, and check the .status attribute when you need to know whether the server was happy.

A minimal request looks like this: import urllib.request; response = urllib.request.urlopen('https://example.com'); html = response.read().decode('utf-8'); print(response.status). One detail that catches newcomers is that urlopen raises URLError or HTTPError for bad URLs, 404s, or network blips, so wrap it in a try block and skip rather than crash. For sites behind Australia's common geo-restrictions or paywalls, you may also need to set request headers manually.

If headers are required, build a urllib.request.Request first and pass it to urlopen. Worth setting are a descriptive User-Agent — bare Python clients get blocked by some servers — and an Accept-Language such as 'en-AU' if you want the Australian variant of a page where one exists.

Extracting Links from the Response

Fetching the page is only half the job. A crawler needs to know where to go next, so it has to parse the HTML for anchor tags. The standard library ships html.parser, and while BeautifulSoup is friendlier, it isn't necessary for a small script. Subclass HTMLParser, override handle_starttag, and grab every 'a' tag, pulling the 'href' attribute out of the supplied attrs list.

Once you have a list of href values, most of them will be relative paths like '/about' or 'contact.html'. Use urllib.parse.urljoin to turn them into absolute URLs against the page you just fetched. Without this step, your crawler will silently try to request '/about' as if it were a complete address and fail. Filtering out non-http schemes such as mailto, javascript, or tel is worthwhile at this stage.

Normalising URLs as you collect them pays back later. Strip fragments after the '#', lowercase the host, and collapse trailing slashes so the visited set doesn't treat '/about' and '/about/' as separate pages. Small details like these are what separate a crawler that quietly does the right thing from one that doubles up on requests and gets throttled by an Australian CDN.

Tracking Visited URLs with a Queue

To stay within bounds, the crawler needs two data structures: a set for URLs it has already seen, and a queue for URLs it still intends to visit. A collections.deque with append and popleft gives you efficient FIFO behaviour for a breadth-first crawl. Each entry in the queue should be a tuple of (url, depth) so the depth counter survives every hop.

When you pop a URL off the queue, first check whether it's already in the visited set. If it is, discard it. If not, fetch it, parse it, mark it visited, and add its outbound links to the queue with depth + 1. This pattern keeps memory usage predictable even on modestly large sites and is the same shape that production crawlers like Apache Nutch and Heritrix use, just stripped down for a single script.

For runs that outlive a single Python process, persist the visited set between invocations. A flat JSON file or a small SQLite table is usually plenty. Without that step, every run starts from zero and the crawler can quietly duplicate work each time it executes.

Enforcing the Depth Limit

The depth cap is the simplest safeguard but also the easiest to get wrong. The most common mistake is checking depth only at fetch time, which means a single page at depth 4 can still spawn fifty links at depth 5. Instead, check the depth when you enqueue new URLs and refuse to add anything that exceeds max_depth. If max_depth is 2, the crawler grabs the seed page, its direct children, and the grandchildren — nothing beyond.

For Australian sites, depth 2 or 3 usually captures everything you need from a small business or council website without stepping outside what the ACCC would consider reasonable. If you're crawling government data sets on data.gov.au, deeper limits may be appropriate because the content is explicitly published for redistribution, but for commercial .au domains it pays to err on the shallow side.

Politeness, robots.txt, and Rate Limiting

A crawler without manners is a fast way to get your IP blacklisted. urllib ships with urllib.robotparser, which parses robots.txt files and tells you whether you're allowed to fetch a given path. Always check it before crawling, even on small personal projects, because habit is what carries you through to bigger ones.

Beyond robots.txt, a short sleep between requests goes a long way. time.sleep(1) before each fetch is enough for most small sites and is well below what an Aussie hosting provider like VentraIP or Aussie Broadband will start throttling on. Setting a descriptive User-Agent string that includes your email address is considered good form, and is required by some .au sites that publish crawler policies. If a site asks you not to crawl it, don't — being a good citizen on the way up pays back later.

Putting a Bounded Crawler to Work

A bounded crawler is handy for far more than scraping headlines. Researchers use them to build local mirrors of public-interest archives, real estate watchers track Sydney and Brisbane listings as they appear, and educators gather primary source materials for classroom use. A working example of an educational source archive shows just how much a focused crawl can uncover when it stays within scope.

What makes this style of crawler pleasant to maintain is that it scales with intent rather than infrastructure. A weekend project can scan a couple of university sites for new publications without anyone provisioning extra servers. Switching to a deeper parser such as lxml speeds things up when a single page carries thousands of links, and persisting the queue to SQLite means a crash mid-run does not undo hours of work. Run the whole thing on a schedule and you have a quiet research assistant ticking along, finishing its sweep before most of Sydney has poured its morning coffee.