How I created a Python script to monitor and log changes in /etc

/etc rarely changes dramatically, yet a small edit can alter SSH access, name resolution, scheduled jobs, or service startup. Package upgrades and administration tools also touch it in passing. I wanted those changes visible without installing a heavyweight security suite.

I built the watcher after a manual audit left me guessing when a resolver file had changed. It captures a baseline, listens for Linux filesystem events, and writes structured records to a file outside the directory being watched.

Python handles paths, timestamps, hashes, deduplication, and JSON. On Linux, inotify-tools supplies the event stream. That combination keeps the program portable across distributions while avoiding repeated full-directory scans during normal operation.

The design suits a modest Linux host in a Melbourne office or a fleet maintained by a small Sydney organisation. It does not replace configuration management or host-based intrusion detection; it supplies the local evidence that makes their activity easier to explain.

Why /etc needs a local audit trail

Changes to /etc often matter after a reboot, when a newly loaded service exposes the result of an earlier edit. A record of the path, old metadata, new metadata, and event type can reveal whether a package manager, administrator, or unexpected process caused the change.

A local trail also survives an internet outage. For an Australian operator working across Canberra and regional offices, a timestamped file on the host is often quicker to inspect than a dashboard on another continent. UTC keeps that record unambiguous while local conversion remains available.

Building the baseline

Before listening for events, the script walks /etc without following symbolic links. It records each regular file’s relative path, mode, owner, group, size, and modification time. Socket, device, and temporary files are skipped.

For an explicit allowlist of non-secret configuration files, it also stores a SHA-256 digest. Password databases, private keys, and files that may contain credentials are never copied or hashed. This gives the baseline enough detail to detect content changes without creating another sensitive store.

The first snapshot is written atomically to /var/lib/etcwatcher/baseline.json. Keeping state under /var/lib rather than /etc prevents the watcher from generating events about its own output, while the --root option lets tests use an ordinary temporary directory.

Connecting Python to inotify

The watcher launches inotifywait with an argument list, not a shell command, and requests create, delete, modify, attribute, and move events recursively. A timeout is avoided so the child process can run as a long-lived service while Python reads its output line by line.

Each event is normalised into a small dictionary containing time, event, and path. Optional fields carry the previous and current mode, ownership, size, modification time, or digest. json.dumps escapes unusual filenames safely, and flush=True makes each record immediately available to less, journalctl, or a collector.

Linux may emit several modify events for one write, so identical records are collapsed within a short debounce window. Periodic reconciliation compares the live tree with the saved baseline, repairing missed events after queue pressure and catching changes made while the process was restarting.

Choosing useful log events

Two short lists keep the raw stream focused on configuration changes that commonly affect access and service behaviour:

• /etc/passwd, /etc/group, and /etc/sudoers
• /etc/ssh/ and firewall configuration directories
• systemd unit directories and cron or timer definitions
• resolver, mail, and automatically renewed TLS configuration

The event field distinguishes actions that can look similar in a file listing:

• a file or directory was created
• content or permissions changed
• an atomic save replaced one path with another
• a watched path was deleted or moved away

Paths can be configured as inclusions or exclusions rather than hard-coded policy. This matters on Debian-family and Red Hat-family hosts, which organise packages, services, and resolver files differently.

Filtering noise without hiding edits

A clean design does not mean a silent log. Generated resolver files, the dynamic linker cache, editor backups, and services that rewrite temporary state can create steady churn. I classify those as low priority while retaining the record, allowing an incident review to ignore them without pretending they never occurred.

Some files need extra care because even their names can disclose system structure. The service uses the minimum capability needed to traverse protected directories, writes only beneath /var/log/etcwatcher, and applies systemd sandboxing. Log files use mode 0640, while private material such as /etc/shadow and SSH host keys receives metadata events only.

The same preference for concise evidence shaped my bot traffic analysis, which uses Scratchy to separate automated and human-looking Apache requests before drawing conclusions from raw logs. Both projects begin with noisy events and reduce them to facts an operator can verify.

Testing the watcher safely

I test against a fixture directory containing nested folders, symlinks, spaces in filenames, and unreadable files. The sequence creates a file, appends to it, changes its mode, performs an atomic rename, and removes it. Each action must produce the expected event without recording the fixture’s own log.

Time handling has its own tests. Log records use an aware UTC datetime, while conversion through zoneinfo must remain correct in Australia/Sydney and Australia/Adelaide, including daylight-saving transitions. This avoids the duplicated or missing local hour that can make Australian incident timelines confusing.

I also test child-process failure, malformed output, a deleted watch descriptor, and inotify queue overflow. On a bad exit, the service returns a failure state for systemd to restart, and the next baseline comparison reveals any gap.

Running the watcher on Australian hosts

The finished systemd unit starts after local filesystems are available, runs with a restricted view, and writes JSON Lines under /var/log/etcwatcher. Logrotate closes and compresses the file without requiring the watcher to reconnect to a remote service. Configuration, sample tests, and licence details accompany the source on the project page.

For teams in Sydney, Melbourne, and Canberra, UTC timestamps provide a common operational clock while operators can render AEST or ACST locally. That separation also helps when an Australian support team coordinates an incident with an overseas hosting provider or distributed development team.

For small organisations in the Australian market, a frequent trigger is certificate renewal for a .com.au website or mail service. The resulting /etc change is recorded beside SSH, sudo, resolver, and service edits, producing a dependable history that remains available after the deployment window closes.