Parsing Binary Logs with Python’s struct Module

Binary logs can look intimidating when a text editor shows a mess of control characters, null bytes and apparently random values. In practice, many formats are carefully organised records: a timestamp, an event code, a length field and a payload, repeated until the file ends. Python gives us enough control to decode that structure without writing a large extension module.

I used the struct module for this kind of work because it makes byte-level parsing explicit. It handles fixed-width integers, floating-point values, byte strings and platform-independent layouts while keeping the implementation readable. That matters when a diagnostic utility may eventually run on Linux servers in Sydney, Melbourne or Brisbane.

The same approach fits naturally with the sort of practical open-source development documented on Phil Schwartz’s portfolio. Log analysis, security utilities and debugging tools often begin with a narrow parser, then grow into dependable command-line programs with filters, reports and useful failure handling.

The examples below use a small invented binary log format, but the principles apply to packet captures, application telemetry, embedded device records and archived system events. The important work is discovering the format, documenting assumptions and making malformed input harmless rather than mysterious.

Start with the binary layout

Before writing Python, I describe each record as a field map. Suppose every entry contains a four-byte Unix timestamp, a two-byte event identifier, a two-byte payload length and a payload containing that many bytes. The corresponding layout is:

timestamp: 4 bytes
event_id:  2 bytes
length:    2 bytes
payload:   length bytes

This gives the fixed header a size of eight bytes. A format document may specify little-endian values, in which case the struct format string is <IHH: < selects little-endian byte order, I represents an unsigned four-byte integer, and each H represents an unsigned two-byte integer.

import struct

HEADER = struct.Struct("<IHH")

Using a compiled Struct object is clearer than repeating struct.unpack() throughout the parser. It also provides HEADER.size, which avoids hard-coding the header length in several places.

Read exact chunks from the file

Binary parsing depends on receiving the exact number of bytes requested. A normal read() call can return fewer bytes at the end of a file, or when data comes from a stream. A small helper makes that condition visible:

def read_exact(stream, size):
    data = stream.read(size)
    if len(data) != size:
        raise ValueError(f"short read: expected {size}, got {len(data)}")
    return data

The parser can then process records in a loop:

def records(path):
    with open(path, "rb") as stream:
        while True:
            header = stream.read(HEADER.size)
            if not header:
                break
            if len(header) != HEADER.size:
                raise ValueError("truncated record header")

            timestamp, event_id, length = HEADER.unpack(header)
            payload = read_exact(stream, length)
            yield timestamp, event_id, payload

Opening the file with "rb" is essential. Text mode can translate line endings and apply encoding behaviour that has no place in a binary format. On Linux this mistake may remain hidden, while moving a tool between development machines and an Australian production environment can expose it quickly.

Make byte order an explicit decision

Byte order is one of the easiest sources of plausible but incorrect data. If a timestamp is stored as 01 00 00 00, little-endian interpretation gives one second after the Unix epoch. Big-endian interpretation gives 16,777,216 seconds. Both are valid integers, so the parser may continue without raising an error.

The struct prefix should therefore come from the format specification, not from the computer running the script. < means little-endian, > means big-endian, and ! selects network byte order. The @ prefix uses native alignment and sizes, which is rarely appropriate for a file format intended to travel between systems.

Alignment deserves similar care. A native C structure may insert padding between fields, but a compact binary log often does not. Using standard-size prefixes such as < or > prevents platform alignment rules from silently changing the expected record size.

Decode payloads according to event type

A length field tells us where the payload ends, but the event identifier tells us how to interpret its contents. For example, event 1 might contain a UTF-8 username, while event 2 might contain a two-byte port number and a four-byte IPv4 address.

def decode_event(event_id, payload):
    if event_id == 1:
        return {"kind": "login", "user": payload.rstrip(b"\0").decode("utf-8")}

    if event_id == 2:
        if len(payload) != 6:
            raise ValueError("network event has an invalid payload size")
        port, address = struct.unpack("<HI", payload)
        return {"kind": "connection", "port": port, "address": address}

    return {"kind": "unknown", "raw": payload.hex()}

Avoid decoding every byte string as text. A binary payload may contain compressed data, a checksum, a packed address or another nested structure. Keeping unknown values as hexadecimal gives an operator something inspectable without corrupting information through a failed character conversion.

Validate lengths before allocating

A corrupt length field can claim that a record contains several gigabytes. Blindly reading that amount is a poor choice for a command-line utility, especially when logs are collected automatically from internet-facing services. Set a reasonable maximum and reject impossible values early.

MAX_PAYLOAD = 1024 * 1024

if length > MAX_PAYLOAD:
    raise ValueError(f"payload is too large: {length} bytes")

A parser should also decide what to do with trailing bytes, unsupported versions and invalid event identifiers. For forensic work, reporting the offset of the bad record is valuable. Track the stream position before reading each header so an error can identify the exact location rather than simply saying that the file is malformed.

Use memory views for larger files

For small logs, reading one record at a time is straightforward and efficient. For large archives, mmap can provide convenient random access without loading the complete file into Python memory. A memoryview can also reduce copying when slicing a byte buffer.

def parse_buffer(data):
    offset = 0
    while offset + HEADER.size <= len(data):
        timestamp, event_id, length = HEADER.unpack_from(data, offset)
        offset += HEADER.size

        if length > MAX_PAYLOAD or offset + length > len(data):
            raise ValueError(f"invalid payload at offset {offset}")

        payload = data[offset:offset + length]
        offset += length
        yield timestamp, event_id, payload

    if offset != len(data):
        raise ValueError("trailing partial header")

unpack_from() is useful here because it reads directly from a buffer at a chosen offset. This style works well for batch analysis of archived logs, including data copied from remote hosts over a slower connection or stored in a local operations team’s evidence directory.

Turn decoded records into useful reports

Parsing is only the first layer. Once records become dictionaries or small data classes, ordinary Python tools can group events, convert Unix timestamps and produce CSV or JSON output. Keeping decoding separate from reporting also makes the parser easier to test.

from datetime import datetime, timezone

def as_dict(timestamp, event_id, payload):
    item = decode_event(event_id, payload)
    item["time"] = datetime.fromtimestamp(
        timestamp, tz=timezone.utc
    ).isoformat()
    return item

UTC is a sensible internal representation for distributed systems. A report intended for operators in Perth, Adelaide or Canberra can convert times to local zones at the presentation layer. This avoids confusing daylight-saving changes in places such as Sydney and Melbourne with actual changes in the source data.

Test the parser with crafted bytes

Binary parsers benefit from tests that construct records rather than relying only on large sample files. struct.pack() creates deterministic fixtures:

header = HEADER.pack(1700000000, 1, 5)
record = header + b"admin"

Tests should cover an empty file, one valid record, multiple records, a partial header, a truncated payload, an oversized length and unknown event types. Include both byte orders if the tool may support more than one format. These cases catch offset errors that ordinary happy-path testing misses.

When the format is undocumented, compare parser output against a known-good producer or a hex editor. Record the observed byte offsets and field values in comments or a format note. That small amount of documentation can save hours when a future version adds a header, changes a counter width or introduces a version flag. The struct module then remains what it should be: a precise bridge between raw bytes and dependable software.