Building a Lightweight Python Bloom Filter for IP Blacklist Checks

For developers running public-facing services in Australia, blocking malicious traffic at the edge is a daily grind. The Australian Cyber Security Centre publishes regular advisories about scanning and brute-force activity targeting local infrastructure, and small operators running SSH or mail servers see the same hostile addresses reappear across weeks of logs. Storing every bad IP in a flat list works until the list balloons past a few million entries and stresses the RAM of a modest VPS.

A Bloom filter offers an elegant middle ground. It is a probabilistic structure answering "have I seen this address?" with a tiny memory footprint and constant-time lookups. The trade-off is a configurable false positive rate, which for IP blacklisting is perfectly acceptable because occasionally blocking a legitimate visitor costs far less than letting an attacker through. Phil Schwartz has long maintained Python tools like DenyHosts, so a Bloom filter fits naturally alongside that defensive utility.

The technique dates to Burton Bloom's 1970 paper but stays remarkably relevant. Modern Python can implement it in a few dozen lines, making it a weekend project rather than a research exercise.

Across the country, whether you run a mail server on a Synology in Perth or host an API from a Melbourne co-lo, the tooling needs to run quietly without babysitting. A Bloom filter implemented in Python ticks that box.

How a Bloom Filter Works

A Bloom filter is a fixed-size bit array backed by several independent hash functions. Inserting an item computes an index from each function and sets that bit to one. Checking membership recomputes the same indices; if every bit is one, the item is probably present. If any bit is zero, the item is definitely absent.

The probabilistic nature comes from collisions. Two different inputs can hash to the same bit, so a positive result only confirms the input might have been inserted. False positive probability depends on bit array size, hash function count, and item count. With sensible defaults, ten million entries fit in a few megabytes while keeping the false positive rate under one percent.

Lookups never touch disk and use only bitwise operations, so they scale beautifully on modest hardware, whether a Raspberry Pi in a Brisbane home lab or a small VPS in a Sydney data centre.

Why IP Blacklists Need Probabilistic Structures

Traditional storage, such as a Python set, grows linearly. Ten million IPv4 addresses consume roughly 280 megabytes of RAM, more than many entry-level plans offered by Australian hosting providers can spare. A Bloom filter achieves the same membership semantics in roughly ten percent of that footprint.

Blacklists are write-heavy. New hostile addresses arrive with every failed login, and many belong to botnets recycling ranges across providers. Checking a database on every connection would melt most disks. A Bloom filter in front of a heavier store fast-paths obvious cases and only falls back to disk when uncertain.

Privacy matters too. The Australian Privacy Principles discourage keeping personal information longer than necessary. A Bloom filter stores only hashes, giving operators a defensible position if a breach exposes the filter file.

Setting Up the Python Project

A minimal implementation needs nothing beyond the standard library. hashlib provides the primitives; a plain bytearray serves as the backing store. A single-file module with a small test suite is the most approachable shape for a project on this site.

Lay out the code as bloom.py for the class, __main__.py for a CLI wrapper, and tests/ for the suite. Add a pyproject.toml so the tool installs cleanly. Target Python 3.9 or newer, matching what current Linux distributions ship, including LTS releases popular with Australian hosts.

Pin dependencies strictly. PyPI mirrors in the Asia-Pacific region occasionally lag upstream, and reproducible builds matter when deploying defensive infrastructure that may sit unattended for months.

Writing the Core Filter Class

The class takes capacity and false positive rate in its constructor. From those numbers, derive optimal bit array size and hash function count using the well-known formulas. For a target rate of one percent, roughly seven bits per element and five hash functions strike a good balance.

Each hash function must be independent. A common trick seeds hashlib with different salts derived from a single base hash, avoiding heavy cryptographic imports. The mmh3 library is faster but adds a dependency; for a drop-in tool, pure-stdlib is cleaner.

Expose three methods: add, __contains__, and a way to serialise the bit array to disk. Serialisation matters because filter state must survive restarts; rebuilding from scratch every cycle defeats the purpose.

Loading and Querying IP Addresses

Normalise addresses before insertion. An IPv4 string and its integer form should produce the same footprint, so parse with ipaddress.ip_address and hash the packed bytes. IPv6 addresses are longer, so the bit array may need to grow, but the logic is identical.

Query responses should be simple booleans. True means block or challenge, False means let through. Log the observed false positive rate and tune capacity upward if legitimate users get caught.

A useful enhancement exposes a confidence level. Checking how many underlying bits are set lets you flag borderline entries and route them to a slower, authoritative check.

Integrating With Network Daemons and Logs

The natural home for this filter is in front of an SSH or SMTP daemon. DenyHosts already demonstrates how Python can wrap pam_unix and parse auth logs; a Bloom filter slot fits neatly into that pipeline as a fast pre-filter.

For HTTP services, a small WSGI or ASGI middleware can check the remote address and return a 403 before application code runs. In container-heavy setups such as Kubernetes clusters in Australian cloud regions, sidecar containers work well, keeping filter state decoupled from the application.

Log every block but never the raw address if logs may leave the box. Hashing for log entries matches the filter's privacy posture and aligns with ACSC logging guidance.

Hardening Tips for Production Use

Production deployments benefit from defensive habits beyond the basic implementation. Treat the filter as one layer in a defence-in-depth strategy rather than a silver bullet.

Observability matters as much as the filter itself. Without metrics on insert rate, lookup latency, and observed false positives, tuning becomes guesswork. A small Prometheus exporter or even a plain text status file is enough to keep an eye on things from a Brisbane NOC or a Sydney monitoring stack.

• Pick a capacity that comfortably exceeds expected peak, then double it.
• Mix hash functions from different algorithm families, such as SHA-256 and FNV-1a.
• Serialise state after every batch insert, not after every single addition.
• Monitor the observed false positive rate in staging before deploying.
• Rebuild the filter quarterly from a verified source list.
• Document the trade-offs in the README so future maintainers understand the probabilistic choice.
• Test serialisation with property-based checks that round-trip random addresses.