Monitoring DNS query latency with a Python watchdog script

When a browser tab hangs for several seconds before showing a page, the bottleneck is rarely the website itself. Name resolution often bears the blame, because every URL has to be translated into an IP address before any content can travel. A sluggish resolver multiplies that delay across dozens of requests per session, leaving users staring at blank screens. Tracking these round-trip times turns an invisible failure mode into a measurable signal that can be acted upon.

Australia adds its own quirks to this picture. Local traffic frequently transits through international gateways operated by Telstra, Optus or TPG, while academic users rely on AARNet for high-throughput links. Geographic distance to authoritative servers in North America and Europe means a typical exchange already runs around 200 milliseconds, so additional jitter is felt sharply. Monitoring DNS performance locally is therefore less an academic exercise and more a practical defence against intermittent slowdowns that affect operations in Sydney, Melbourne and beyond.

Python is well suited to this kind of background work. The language ships with timers in the time module, statistical helpers in statistics, and straightforward networking through sockets and third-party packages. Phil Schwartz, whose portfolio of open-source utilities includes DenyHosts, Kodos and Scratchy, has long demonstrated how a few dozen lines of Python can replace commercial tooling. The same ethos applies here: a compact script can quietly record resolver behaviour around the clock without consuming meaningful resources.

What follows walks through a minimal yet production-ready script that measures lookup latency, archives the readings, raises warnings when figures drift out of bounds and runs unattended on a Linux host. It assumes a working knowledge of Python 3 and basic shell commands, but nothing more exotic than that.

Laying out the Python toolchain

A clean virtual environment keeps the project tidy and avoids clashes with system packages. Most Australian developers run Ubuntu or Fedora on their workstation, where the standard pattern is python3 -m venv .venv followed by source .venv/bin/activate. Inside that environment, the only external dependency worth pulling in is dnspython, which exposes a friendly wrapper around the low-level resolver calls and returns structured records instead of raw bytes.

Pinning versions in a small requirements.txt makes the script reproducible across laptops, dev servers and a Raspberry Pi sitting under a desk in Adelaide. A line such as dnspython==2.6.1 guarantees identical behaviour when the tool is later deployed to a VPS in Sydney or Singapore. Keeping the dependency surface small also simplifies auditing, which matters for anyone working under the Australian Cyber Security Centre's Essential Eight guidance.

Writing the measurement function

The heart of the script is a function that accepts a hostname and returns a duration in milliseconds. A straightforward implementation uses dns.resolver.Resolver from dnspython and time.perf_counter to bracket the call. Storing the result alongside a UTC timestamp produces a record that is easy to plot later and easy to correlate with incident reports from other monitoring tools.

Running each query several times and keeping the median reading helps, because retransmissions and resolver quirks can otherwise inflate the numbers. The statistics module provides mean, median and stdev for quick summaries, although a sorted list of recent samples is often more useful for spotting creeping degradation. Choosing a small but representative set of targets such as google.com.au, aarnet.edu.au and a couple of internal .com.au domains gives a realistic cross-section of Australian traffic.

Logging and data storage

Persisting the readings allows trends to emerge over weeks rather than minutes. A SQLite database, accessed through the standard sqlite3 module, handles millions of rows on a Raspberry Pi without breaking a sweat and avoids running a separate database server. A single table with columns for timestamp, hostname, query type, duration and result code keeps the schema readable and the indexes cheap.

For those who prefer flat files, appending to a daily CSV works just as well. Rotating the file with a small helper based on logging.handlers.RotatingFileHandler mirrors the approach used by tools such as Scratchy on this site. The script can then be wired into log shipping stacks that already ingest Apache or Nginx output, which is convenient for administrators in Brisbane or Perth data centres.

Triggering alerts when latency spikes

Detection is only useful if someone hears about it. The simplest path is a function that compares the latest reading against a configurable threshold and, when exceeded, fires a notification. For local operators, an email dispatched through smtplib to a distribution list often suffices. Larger shops may prefer a webhook posted to Slack, Microsoft Teams or PagerDuty, all of which accept a JSON payload.

Alerts should carry enough context to be acted upon without opening the script: the hostname queried, the measured duration, the threshold breached and a short history of recent readings. Following ACSC advisories on clear, actionable alerting helps avoid the fatigue that sets in when notifications fire on every minor blip. A short back-off, such as one alert per incident every fifteen minutes, also reduces noise during a transient upstream outage.

Running the script continuously on a Linux host

A monitoring tool only earns its keep when it runs unattended. The conventional way to do that on modern distributions is to drop a small systemd unit into /etc/systemd/system/, pointing the ExecStart at the Python interpreter inside the virtual environment. Enabling the unit with systemctl enable dns-watchdog.service ensures it survives reboots, and a Restart=on-failure directive handles the occasional network blip.

Logs land in the systemd journal, where they can be tailed, filtered and forwarded with familiar utilities. Running journalctl -u dns-watchdog -f on a quiet Saturday afternoon in Hobart is a perfectly reasonable way to verify the service is alive. Coupling the unit with a logrotate rule for the CSV output closes the loop, ensuring long-running deployments do not fill the disk.

Calibrating thresholds and avoiding noise

Picking a sensible threshold is partly science and partly local knowledge. A median of 80 milliseconds across the test set is a reasonable starting point for Australian conditions, but operators on NBN fibre to the premises will see lower baselines than those on satellite services in regional Western Australia. Collecting a week of quiet-period data before enabling alerts gives the script a genuine baseline rather than a guessed number.

Percentile-based thresholds, such as alerting when the 95th percentile of the last hour exceeds a target, cope better with brief spikes caused by busy resolver caches. Reviewing the alert log after a fortnight and adjusting bounds based on actual experience is far more productive than chasing a single magic value. The script's small footprint means it can run alongside existing infrastructure without contention, leaving operators free to focus on rare events that genuinely warrant attention.