Crafting a Python Module for Apache Combined Log Format

Australia's web infrastructure keeps humming along in data centres stretching from Perth to Brisbane, and somewhere in nearly every one of those facilities an Apache web server is quietly writing line after line into a combined log file. These logs are a goldmine for understanding traffic patterns, debugging application issues, and meeting obligations under the Privacy Act 1988 when personal data appears in referrer strings or user agent fields. A reliable parser is essential whether you are auditing access for a Melbourne e-commerce site or monitoring a regional council portal in Hobart.

Phil Schwartz's own Scratchy project has long demonstrated how useful an Apache log analyser can be, but there are good reasons to roll a smaller, focused module instead of pulling in a full application. A dedicated parsing module slots neatly into scripts, test suites, or data pipelines, and it gives you full control over how edge cases are handled. The aim here is to build a self-contained Python package that can both read and emit entries in the Combined Log Format, with clean APIs and thorough test coverage.

The module we are about to build will expose two main functions: one for taking a raw line and returning a structured record, and one for converting a structured record back into a properly formatted line. Along the way we will look at the subtleties of the format, handle the messy realities of real log files, and wrap the whole thing up so it can be installed with pip.

Anatomy of the Combined Log Format

Before writing any code, it helps to be precise about what we are parsing. The Combined Log Format is an extension of the Common Log Format that adds two fields at the end: the referrer and the user agent string. A typical line looks something like this:

127.0.0.1 - frank [10/Oct/2023:13:55:36 +1100] "GET /index.html HTTP/1.1" 200 2326 "https://example.com/" "Mozilla/5.0 ..."

The format is deceptively simple, with a handful of quirks worth remembering:

Fields are separated by single spaces, but the request, referrer and user agent are wrapped in double quotes because they can contain spaces.
The timestamp is wrapped in square brackets and includes a timezone offset, which for Australian servers is typically +1000 for AEST or +1100 for AEDT.
The bytes field is either a positive integer or a dash when the response sent zero bytes, as happens with 304 Not Modified replies.
Several fields are conventionally a single dash when the information is not available, so the parser must accept dashes in the ident, user and bytes positions.

These quirks mean that a naive split on whitespace will not work. The standard approach is to use a regular expression that captures each quoted block as a single unit while splitting on whitespace elsewhere.

Setting up the module skeleton

A tidy layout makes the module easy to maintain. Inside a directory called apachelog we create an __init__.py that re-exports the public functions, a parser.py for the parsing logic, a formatter.py for the reverse operation, and a tests folder for unit tests. A pyproject.toml file declares the build system and metadata so the package can be uploaded to PyPI or installed from a local clone.

In parser.py we define a custom exception, ApacheLogError, to signal malformed input, and we declare the regular expression as a module-level compiled pattern. Compiling once and reusing the compiled object avoids the overhead of recompilation for every line, which matters when you are processing a busy production server pumping out millions of entries per day, particularly during end-of-financial-year traffic spikes on Australian retail sites.

The skeleton might look like this:

import re
from datetime import datetime

class ApacheLogError(Exception):
    pass

LOG_PATTERN = re.compile(
    r'(?P<host>\S+) '
    r'(?P<ident>\S+) '
    r'(?P<user>\S+) '
    r'\[(?P<time>[^\]]+)\] '
    r'"(?P<request>[^"]*)" '
    r'(?P<status>\d{3}) '
    r'(?P<bytes>\S+) '
    r'"(?P<referer>[^"]*)" '
    r'"(?P<user_agent>[^"]*)"'
)

The named groups make it trivial to pull individual fields out of a match object.

Implementing the parser function

The parser itself is short. It takes a string, attempts to match it against the compiled pattern, and returns a dictionary if the match succeeds. If the match fails, it raises ApacheLogError with a descriptive message including the offending line number.

A subtle decision is how to represent the timestamp. Keeping it as a string preserves the original formatting, but converting it to a datetime object makes downstream analysis easier. A good compromise is to return both: the raw string for faithful round-tripping, and a parsed datetime for convenience. The format string %d/%b/%Y:%H:%M:%S %z handles the bracketed timestamp reliably, including the Australian timezone offsets mentioned earlier.

Bytes can be either a number or a dash when no bytes were sent. The parser should preserve the raw value while also providing a helper to interpret it as an integer when needed.

Handling real-world log quirks

Production logs are messier than the textbook example. Some fields may contain embedded quotes that have been escaped, log rotation tools may insert partial lines, and a malicious client may try to break the parser with crafted input. A robust module needs to address these realities rather than assume every line is well formed.

Common edge cases worth covering include:

A status code that is not a three-digit number due to a misbehaving upstream proxy.
A request line missing the protocol, producing something like "GET /path" without HTTP/1.1.
User agent strings containing unusual characters from older browsers or scraping bots.
Very long lines that exceed any reasonable buffer if read into memory all at once.

For high-volume use, the parser should accept an iterable of lines and yield records one at a time using a generator. This keeps memory usage flat regardless of file size, which is helpful when replaying months of historical data from a busy Australian retailer during a peak sales period.

Formatting entries back into strings

The reverse operation is occasionally useful for log reformatting, for anonymisation tasks required under the Notifiable Data Breaches scheme, or for generating test fixtures. A format_entry function takes a dictionary (or a small dataclass) and returns a correctly quoted line.

The trickiest part is escaping any double quotes that appear inside fields like the referrer or user agent. Apache uses backslash-escaped quotes inside the quoted blocks, so the formatter must replace each " with \" before wrapping the field in its outer quotes. The timestamp should be re-rendered in the same bracketed format with the original timezone offset preserved, so a log emitted by the formatter is byte-for-byte identical to the original after a round trip.

Packaging, testing, and documentation

A module without tests is a liability. A thorough test suite exercises the parser with representative lines, malformed input, and round-trip cases where a parsed entry is formatted and re-parsed to confirm equality. pytest is the usual choice and runs comfortably on developer laptops from Adelaide to Darwin.

Documentation belongs in the docstrings and a short README. The README should show how to install the module, present a minimal usage example, and note the licence. Given the open-source ethos behind Phil Schwartz's other projects, an MIT or BSD licence is a sensible default that encourages reuse across the Australian open-source community.