Build a Python IP overlap checker for reliable security analysis

Comparing two lists of IP addresses sounds simple: read both files, find values that appear in each, and print the matches. In practice, production data contains blank lines, comments, duplicate entries, IPv6 addresses, inconsistent formatting, and sometimes CIDR network ranges. A small Python utility can turn that messy input into a repeatable security check.

This kind of tool is useful for Linux administrators, incident responders, and developers maintaining open-source security utilities. An Australian hosting provider in Sydney, a university network in Melbourne, or a business using NBN connections in Brisbane may all need to compare firewall rules, blocklists, allowlists, or recent authentication logs.

Define what an overlap means

For two plain address lists, an overlap is an exact address present in both collections. If 203.0.113.8 appears in the first file and the second file, it is a match. The order of the addresses does not matter, and repeated occurrences should usually be counted once.

This definition changes when the input contains networks. The address 192.0.2.15 overlaps with 192.0.2.0/24, even though the strings are different. A first version should clearly support exact IP addresses, while a later version can add CIDR-aware comparisons with Python’s standard ipaddress module.

Choose sets for fast comparisons

Python sets are a natural fit because they store unique values and provide an efficient intersection operation. After converting both input lists to sets, first & second returns the shared addresses. This is clearer and generally faster than checking every item in one list against every item in the other.

The set approach also removes duplicate entries automatically. That matters when parsing Apache logs or firewall exports, where the same hostile address may occur hundreds of times. If frequency is important, retain a separate collections.Counter; use the set intersection for identifying shared values.

overlap = addresses_a & addresses_b

for address in sorted(overlap, key=lambda value: ipaddress.ip_address(value)):
    print(address)

Sorting with ipaddress.ip_address produces sensible numeric ordering instead of placing 10.0.0.100 before 10.0.0.20 as ordinary text.

Parse input without trusting its format

A useful command-line program should accept one address per line and ignore blank lines. Comments beginning with # are convenient for hand-maintained blocklists, while whitespace should be stripped before validation. Invalid values should generate a useful error rather than silently disappearing.

from ipaddress import ip_address

def read_addresses(filename):
    addresses = set()

    with open(filename, encoding="utf-8") as stream:
        for line_number, raw_line in enumerate(stream, 1):
            value = raw_line.split("#", 1)[0].strip()

            if not value:
                continue

            try:
                addresses.add(str(ip_address(value)))
            except ValueError as error:
                raise ValueError(
                    f"{filename}:{line_number}: invalid IP address {value!r}"
                ) from error

    return addresses

Converting each value through ip_address normalises IPv4 and IPv6 notation and validates the input. It also prevents a typo from producing a misleadingly small overlap report.

Add a clear command-line interface

A small CLI makes the checker easy to use in shell scripts, cron jobs, and CI pipelines. argparse provides help text and consistent error handling without adding third-party dependencies, which is valuable for a portable Linux utility.

import argparse
import ipaddress

def main():
    parser = argparse.ArgumentParser(
        description="Print IP addresses present in both input files"
    )
    parser.add_argument("first_file")
    parser.add_argument("second_file")
    args = parser.parse_args()

    first = read_addresses(args.first_file)
    second = read_addresses(args.second_file)
    shared = first & second

    for address in sorted(shared, key=ipaddress.ip_address):
        print(address)

    print(f"Shared addresses: {len(shared)}", file=sys.stderr)

The complete script should import sys for the summary stream and return a useful exit status. Printing matches to standard output keeps the result pipe-friendly, while sending the count to standard error allows commands such as checker allowlist.txt blocklist.txt > matches.txt.

Handle IPv4 and IPv6 deliberately

Australian organisations increasingly operate dual-stack networks, and cloud systems may expose both address families. A checker that assumes every value contains four decimal octets will reject legitimate IPv6 addresses such as 2001:db8::25. Using ipaddress.ip_address avoids that limitation with no external package.

You may still want to report mixed input clearly. Comparing IPv4 and IPv6 values is safe because they cannot be equal, but a diagnostic summary can tell an operator how many addresses of each family were loaded. This is helpful when a Sydney-hosted application has an IPv4 blocklist while its IPv6 traffic is logged separately.

Extend the tool for real security data

Exact matching is appropriate for many blocklist tasks, but network containment requires ip_network and ip_address. A value such as 203.0.113.0/25 represents a range, so the program needs to decide whether an address belongs to that range or whether two networks overlap. Keep this feature separate from exact comparison to avoid surprising results.

Large feeds also benefit from streaming and structured output. JSON or CSV output can be consumed by monitoring systems, while a --quiet option can suppress normal output and use the exit code to indicate whether a match exists. Avoid sending untrusted IP data directly into shell commands; pass values as arguments or process them inside Python.

Test the checker with representative data

Testing should cover normal matches, duplicates, invalid lines, comments, blank input, IPv4, IPv6, and files with no overlap. Include addresses from documentation ranges such as 192.0.2.10 rather than real customer data. A test suite using unittest or pytest will protect the parser when new features are added.

Australian operations often combine exports from different systems: a Perth office firewall, a Melbourne cloud workload, and a managed service in Sydney may use different formatting conventions. Test files should imitate those realities, including Windows line endings and trailing comments copied from an analyst’s notes.

Useful test fixtures

Two files containing the same IPv4 address
Duplicate entries mixed with blank and comment lines
Valid IPv6 addresses alongside IPv4 values
An invalid address with its filename and line number

Operational checks worth automating

A non-zero status when a parsing error occurs
Stable numeric ordering in generated reports
Correct handling of empty files and no matches
A summary count that equals the unique overlap

Keep the utility small and maintainable

A focused script is easier to audit than a large dependency-heavy application. Separate file loading, validation, comparison, and presentation into functions, then add tests for each part. Document whether the program performs exact address matching or CIDR-aware network analysis so users do not mistake one behaviour for the other.

For an open-source project, include a licence, usage examples, and sample input files that contain documentation addresses. That makes the tool suitable for Linux distributions, internal Australian IT teams, and developers adapting it alongside log analysers or SSH protection utilities. Clear output and predictable exit codes will matter more than a complicated interface.