Building a Python-based configurable log file watcher with callbacks

Modern deployments produce a steady stream of log files that need to be watched, parsed, and acted upon. From a small web app on a Linode in Sydney to a fleet of services in a Brisbane data centre, the need to react to log events in near real-time comes up repeatedly. Rather than reaching for a heavyweight tool, many developers reach for a few hundred lines of Python wrapped around inotify or a simple polling loop.

This piece walks through the design of a small, configurable log file watcher that supports custom callback functions. The aim is a tool that drops into any environment — whether that is a developer's laptop in Adelaide or a production cluster in Canberra — and runs within minutes. Configuration is read from a plain file, patterns are defined per watched path, and the response to each match is delegated to whatever Python callable you provide.

Why a custom log file watcher makes sense

Off-the-shelf log shippers like Filebeat or Fluent Bit are excellent, but they carry operational overhead. They expect a running pipeline, a destination service, and a fairly uniform configuration format. For a developer on a side project, or for a sysadmin in Perth supporting a mining firm's intranet tools, the friction of standing up a full ELK stack just to flag a recurring stack trace in /var/log/app.log is often too high.

A lightweight Python watcher fills that gap. It sits in the background, watches a handful of files, and calls back into application code that already exists. The same script that writes audit logs can also react to them. This keeps the dependency surface tiny — typically the standard library plus PyYAML for configuration — and makes the behaviour auditable.

Australian engineering teams often prize pragmatic tools that do one thing well, and a watcher with a clean callback interface matches that ethos. It is not a pipeline, a shipper, or a dashboard. It watches files and calls your code.

Core architecture choices for log monitoring

The two main approaches for watching files on Linux are filesystem notifications via inotify and periodic polling. inotify is more efficient because the kernel tells your process when a file changes. Polling simply re-reads the file on a timer and is portable to macOS and BSDs, which matters when developers are split between Hobart and San Francisco.

A reasonable design uses inotify on Linux and falls back to polling elsewhere. The watchdog library abstracts both, but writing the watcher from scratch with the selectors module is a useful exercise that keeps the code auditable. The watcher can run on a dedicated thread, with a queue feeding matched events to the callback dispatcher.

Handling log rotation is the trickiest part. When logrotate moves /var/log/app.log to /var/log/app.log.1 and creates a new file, a naive watcher can lose its position or duplicate events. Tracking the inode and device number, as tail -F does, solves this. Storing offsets within the rotated file also helps when the watcher resumes after a restart.

Reading and applying configuration files

Configuration drives flexibility. A watcher that only watches one hard-coded path is rarely useful. Each watch target should describe its file or directory, glob pattern, regex for matching new lines, and which callback to invoke.

YAML is a common choice because it is human-friendly and supports nested structures. A configuration block might specify encoding, start_at_end, cooldown_seconds, and a list of rules, each with its own pattern and callback name. A registry then maps callback names to Python callables, so the configuration stays declarative and the code stays in the repository where it can be reviewed.

Time zones deserve attention in Australia. Servers in Sydney and Melbourne run on AEST or AEDT depending on daylight saving, while Brisbane stays on AEST year-round and Perth sits on AWST. A log line timestamped during the daylight saving crossover can confuse simple parsers, so storing timestamps in UTC internally and converting at the boundary keeps things sane. The configuration file is a natural place to specify the desired display zone.

Wiring up callbacks for flexible responses

Callbacks are where the watcher becomes useful. A callback receives a structured event — the matched line, the file path, the rule that fired, and any named capture groups — and decides what to do. Common responses include writing to a database, posting to a chat channel, sending an email, or invoking an HTTP endpoint.

Decoupling the watcher from the response means the same tool is reused across very different scenarios. A team in Adelaide supporting an e-commerce site might use it to alert on 500 errors in nginx logs. A research group in Hobart monitoring ocean buoy telemetry might use it to flag malformed CSV rows. The watcher code does not change; only the callbacks and configuration do.

Debouncing and rate-limiting matter when a single log event can cascade. If a faulty upstream service starts emitting thousands of stack traces per second, the callback should be able to throttle itself. Adding a cooldown_seconds field per rule, or a token bucket at the dispatcher level, prevents the watcher from amplifying a problem into a denial of service. During the EOFY crunch, when Australian finance teams push batch jobs through the night, this guardrail becomes especially valuable.

Running, testing, and deploying the watcher

Running the watcher as a long-lived process under systemd is the simplest deployment. A unit file pointing at the Python entry point, with Restart=on-failure and a quiet log level, is usually enough. For containers, a small Docker image based on python:3-slim keeps the image size manageable, which matters when pushing builds over a metered NBN link from a regional office.

Testing the watcher is straightforward because the callbacks can be swapped for fakes. Write a test that creates a temporary file, appends lines, and asserts that the registered callback receives the expected event objects. Property-based testing works well here — fuzz the watcher with random writes and rotations and confirm no exceptions escape.

For observability, the watcher should expose its own metrics: files watched, events processed per second, last seen timestamp per file. Exposing these over a simple HTTP endpoint or pushing to a Prometheus gateway fits the patterns already common in Australian teams who follow ACSC guidance for critical infrastructure. A watcher that watches other systems should itself be watchable.