Verifying file integrity with Python's hashlib after transfer

When a software package travels from a developer's laptop in Berlin to a CI runner in Sydney, the bytes that arrive are not guaranteed to match the bytes that left. Network equipment drops packets, intermediate caches rewrite headers, and the long trans-Pacific route most Australian downloads take introduces more opportunities for silent corruption than a short hop. A single flipped bit deep inside a multi-gigabyte image can render an installation unbootable, and the user has no obvious way to know what went wrong. This is why cryptographic fingerprints have been part of every serious distribution workflow since the earliest Linux ISOs shipped with MD5SUMS files.

Python's standard library has included the hashlib module for decades, and it remains the most practical way to generate and compare digests without extra dependencies. The module wraps OpenSSL's optimised implementations of SHA-2, SHA-3, BLAKE2 and the older algorithms that refuse to disappear. For most developers the value lies in predictability: the same input always produces the same output, the output is short enough to paste into a forum post, and comparing two digests is faster than re-reading a large tarball.

The technique is not new, but it is underused in projects where it would save the most trouble. Open source maintainers distributing tarballs, photographers moving RAW files between backup drives, and scientists shipping climate datasets between universities in Brisbane and Perth all benefit from a small verification step. The rest of this article walks through choosing an algorithm, writing a verifier, and weaving the check into scripts that run unattended on remote servers.

The role of checksums in everyday file movement

Data integrity is a property of the bits themselves, not of the transfer protocol. Even with TLS, the application layer can mangle content through truncation, encoding mistakes, or bugs in custom code. A checksum is a short, fixed-length value derived from the original payload through a one-way function, and any change to the input — even swapping two bytes — produces a different output. That asymmetry is what makes the digest a useful tamper detector.

In Australian practice, this matters more than the global average. Household connections through the National Broadband Network saturate during evening peak hours, occasionally leaving files with sparse corruption that passes casual inspection. Remote workers hopping between Melbourne co-working spaces and home offices in regional Victoria also rely on USB drives and SSDs that develop bad blocks over time. A quick digest comparison catches both classes of failure before they cascade into lost work.

For open source distribution, the convention has hardened around a small set of files placed next to the download: a SHA256SUMS text document and a detached signature. Users fetch the binary, recompute the hash locally, and confirm the string matches. The workflow is simple, but it requires a small piece of glue, and Python's hashlib provides the primitives needed to write it.

Choosing an algorithm that fits the threat model

The module exposes a factory function, hashlib.new, that takes a string identifier such as "sha256", "sha512", "blake2b" or "md5". Each variant has different construction, output length, and resistance to known attacks. MD5 and SHA-1 are useful for compatibility with older manifests, but they should never be the sole check where authenticity is at stake, because collision attacks against them are well documented.

For most contemporary work, SHA-256 strikes a reasonable balance. It produces a 64-character hexadecimal digest, runs in constant memory, and is supported by every operating system that can run Python 3. SHA-512 is faster on 64-bit hardware and is worth considering for very large files. BLAKE2 is often quicker than SHA-2 at the same security level and is built into hashlib without extra configuration.

A maintainer publishing a release for the Australian open source community can rely on SHA-256, since the tools ship with every Linux distribution, macOS, and modern Windows build of Python. If the project must interoperate with hardware that only understands MD5, the older algorithm is acceptable for non-sensitive builds, provided the manifest also offers a stronger alternative.

Building a simple verifier

The most common script fits on a single screen. The general shape: open the file in binary mode, feed its contents into the hash object, and read the hexadecimal digest at the end. A minimal version, adapted from a helper used by the canyonero project, looks like this:

import hashlib

def digest(path, algo="sha256", chunk=1024 * 1024):
    h = hashlib.new(algo)
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

The function reads the file in one-megabyte blocks rather than slurping the whole thing into memory, which keeps it usable on the smallest cloud instances and on a Raspberry Pi in a server rack in Adelaide. The iter call with an empty bytes sentinel is a standard idiom for looping until read returns nothing.

Once the digest is computed, comparing it against an expected value is a string equality test. For better diagnostics, printing both values with a clear label makes CI logs easier to read, sidestepping the silent success problem where a truncated read returns an empty digest that happens to match an empty manifest entry.

Handling large files and slow links

Memory pressure is the silent failure mode of naive verification scripts. A snippet that calls path.read_bytes() on a forty-gigabyte VM image will swap itself to death on a workstation with sixteen gigabytes of RAM. The chunked approach above avoids that, and a progress indicator helps when downloading a dataset over a slow connection during peak hours.

For very large transfers, BLAKE2b in tree mode is worth knowing about. The constructor accepts a tree_hashing keyword that lets callers update the hash in parallel across threads, scaling linearly with core count. The Australian Synchrotron publishes multi-terabyte diffraction datasets whose checksums are produced in parallel before the files are mirrored to international repositories.

Another practical consideration is the encoding of the expected digest. Some projects publish raw bytes in base64, others lowercase hex with or without a sha256: prefix. Stripping prefixes, lowercasing both sides, and decoding base64 where needed turn a brittle script robust. The Python bytes.fromhex and base64.b64decode helpers handle both formats without external libraries.

Wiring verification into automated workflows

Verification earns its keep when it happens without a human in the loop. A Makefile target that downloads an archive, recomputes its SHA-256, and aborts the build if the value does not match is more reliable than a checklist taped to a monitor. The same script can refresh nightly mirrors, which is how community-run Australian archives keep their content trustworthy without paid staff.

Container builds benefit from the same approach. Pinning a base image by digest in a Dockerfile turns the tag into an immutable reference, and a pre-build script asserts the pulled layer matches the digest in version control. If a registry is compromised, the build fails before any compromised image reaches a cluster.

Storing digests in a log file alongside timestamps and the script version that produced them gives future maintainers a paper trail. When a user reports a corrupted file years later, a grep against the log often confirms whether the issue sits with the source, the mirror, or the local copy. The investment in a few lines of hashlib code keeps paying back long after the transfer.