Advertisement
July 17, 2011
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
Kodos — Python Regex Debugger
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
DenyHosts — SSH Attack Blocker
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
Scratchy — Apache Log Analyzer
Blue and gray watering wand spraying multiple thin streams of water upward over a green lawn
Canyonero — Project Management
Open Source Projects by Phil Schwartz

Writing a Python Daemon for MQTT System Health Metrics

A small Python service can turn an ordinary Linux host into a useful observability source. Instead of waiting for a failure, it can publish CPU load, memory pressure, disk usage, temperatures and uptime to an MQTT broker at regular intervals. A dashboard, alerting rule or home automation system can then consume those readings.

This approach suits anything from a Raspberry Pi on an Australian NBN connection to a virtual machine supporting a production application. MQTT keeps the messages compact, while Python offers mature libraries for reading operating system statistics and handling network communication.

The most reliable design separates measurement from transport. The daemon gathers a snapshot, converts it into predictable JSON, publishes it with a clear topic, waits for the next interval and records useful errors. Running it under systemd is generally safer than writing a custom backgrounding mechanism.

For a developer maintaining Linux utilities, this is also a practical exercise in packaging, permissions, logging and defensive programming. The result should be quiet when everything is healthy, informative when something changes and easy to inspect when a server has a rough arvo.

Choose A Clear Metrics Model

Begin by deciding what the service will publish. A compact payload might include a Unix timestamp, hostname, CPU utilisation, one-minute load average, memory percentage, root filesystem usage and uptime. Temperature is valuable on small computers, although its availability varies by hardware and operating system.

Use a stable topic hierarchy such as systems/brisbane-pi/health or servers/web-01/metrics. A shared prefix makes subscriptions straightforward, while the hostname prevents collisions. If devices move between locations, keep geography in configuration rather than hard-coding it into the Python module.

JSON is a sensible default because it remains readable during troubleshooting and works with most MQTT dashboards. Keep field names consistent and publish numeric values as numbers rather than formatted strings. A consumer should be able to compare memory_percent: 72.4 without parsing a sentence.

Collect Data With Python

The psutil package provides portable access to CPU, RAM, disks, network counters and sensors. A minimal collector can call psutil.cpu_percent(interval=None), psutil.virtual_memory(), psutil.disk_usage("/") and psutil.boot_time(). The first CPU reading may be meaningless, so prime the counter before entering the main loop.

Avoid collecting every available statistic simply because the library exposes it. A lean payload reduces bandwidth and makes dashboards easier to understand. On a fanless device in Perth, for example, temperature and load may be more useful than a long list of interface counters during a hot spell.

Handle missing sensors gracefully. psutil.sensors_temperatures() can return an empty dictionary or raise an exception on a virtual machine. The daemon should publish null, omit the optional field or log a debug message rather than stop sending all other health data.

Useful Core Fields

Configure MQTT Carefully

Eclipse Paho MQTT is a common Python client for publishing messages. Configure the broker hostname, port, topic prefix, client ID and interval through environment variables or a TOML, YAML or INI file. Keeping credentials outside the source tree is essential, particularly when code is stored in a public repository.

For a real deployment, use MQTT over TLS with a username or certificate-based identity. Port 8883 is widely used for encrypted MQTT, while an internal broker may expose another secured listener. Validate the broker certificate instead of disabling verification to make a quick test pass.

Set a modest keepalive and use a persistent client connection when publishing every few seconds or minutes. A last-will message can announce that a device is offline, while a retained health message lets a newly connected dashboard display the latest known state. Retained data should include a timestamp so an old reading is not mistaken for a current one.

Build A Resilient Service Loop

The central loop should collect metrics, serialise the payload, publish it and sleep until the next scheduled run. A simple time.sleep(interval) is adequate for a small utility, though a monotonic deadline avoids drift when collection or network operations take noticeable time.

Catch connection failures and retry with bounded backoff. The service must continue running through a brief Wi-Fi outage, broker restart or NBN drop rather than exit after one exception. Avoid hammering the broker: wait a few seconds, then increase the delay up to a sensible maximum before trying again.

Use signal handlers for SIGTERM and SIGINT, set a shutdown flag and disconnect cleanly. This matters when systemd stops the service during maintenance. Logging should include connection state, publish failures and recovery events, but never passwords or private certificate content.

Run It Under Systemd

A native systemd unit provides supervision, startup ordering and a consistent log destination. Run the daemon as a dedicated unprivileged user with read access only to the resources it needs. Most metrics require no root privileges, so avoid granting elevated access merely because it is convenient.

A service file can use Restart=on-failure, RestartSec=10, EnvironmentFile=/etc/health-mqtt.env and an explicit ExecStart pointing to a virtual environment. Add After=network-online.target and Wants=network-online.target when the broker requires a ready network at boot.

Test the unit with systemctl status and inspect output using journalctl -u health-mqtt. A useful health daemon should explain whether it is collecting data, connecting, publishing or backing off. Older Python utilities and related developer projects can be found among these software projects, which can provide ideas for keeping small tools focused and maintainable.

Secure And Operate The Daemon

Treat MQTT topics as an access-control boundary. Give each device a client identity and permit it to publish only below its own topic prefix. Dashboards and alerting services should receive subscriptions without gaining publish access to every system.

Be cautious with labels that reveal infrastructure. A hostname such as billing-prod may expose more than necessary if messages leave a private network. Australian organisations may also need to consider where broker data is hosted, especially when operational telemetry could be linked to customers or regulated services.

Choose an interval based on the job. Thirty seconds may suit a temperature experiment, while five minutes is enough for disk capacity and general server health. For sites spread between Sydney, Melbourne and Darwin, use UTC timestamps in payloads and convert them in the dashboard; Australian daylight saving rules vary by state.

Operational Checks

A small alerting layer turns metrics into action. Trigger warnings when a filesystem reaches 80 or 90 per cent, memory remains high for several samples, or no message arrives within twice the expected interval. Temperature thresholds should reflect the hardware and local conditions: a machine in a Brisbane cupboard during a humid summer afternoon may behave differently from one in a cool Melbourne office.

Practical Deployment Values

Keep the code testable by placing collection, payload creation, MQTT publishing and scheduling in separate functions. Mock psutil and the MQTT client in unit tests, then perform one integration test against a local broker. That structure makes future additions—such as network throughput or SMART disk data—less likely to disturb the core service.