Monitoring per-process disk I/O on Linux with Python and /proc

When a Linux server starts thrashing its disks, knowing the culprit process can turn a stressful evening into a quick fix. Whether you run a small Flask app on a VPS in Sydney or process climate data for the Bureau of Meteorology in Melbourne, granular visibility into read and write bytes is invaluable. Python's standard library already exposes everything needed to walk the /proc filesystem.

The kernel publishes per-process accounting through /proc that is rarely surfaced by top or htop. For Australian administrators maintaining research clusters on AARNet or hobbyists tuning home labs over the NBN, this information is gold. A short Python program can poll /proc/[pid]/io at fixed intervals, compute deltas, and present results in a human-readable format.

This article walks through building such a script from scratch. The approach is deliberately low-level: no external libraries, just os, time, and a few lines of standard Python. The result is portable across distros, transparent to audit, and easy to adapt for containers, slow CI runners, or simply curiosity on a quiet Sunday in Adelaide.

Understanding the /proc filesystem

/proc is a virtual filesystem exported by the kernel. Its contents are not stored on disk but generated on demand when a user-space program reads them. Each running process has a directory at /proc/[pid] containing counters for memory maps, file descriptors, and resource usage.

The file of interest is /proc/[pid]/io, which exposes read_bytes, write_bytes, syscr, syscw, and related counters. Crucially, read_bytes and write_bytes count actual I/O rather than buffered transfers, making them useful for spotting processes that genuinely stress the storage subsystem. Where cgroups v2 is enabled, similar data is available through /sys/fs/cgroup, but the per-process view in /proc remains the most universally compatible option.

Values are cumulative since the process started, so a single sample reveals nothing about throughput. Polling at intervals and computing deltas is essential, and some kernels restrict access to /proc/[pid]/io to the owning user or privileged callers, so the script must handle permission errors gracefully.

Reading process I/O counters

Python's open function handles virtual files like regular ones, keeping the implementation simple. Reading /proc/[pid]/io returns a small text payload where each line is a key-value pair. The script needs only read_bytes and write_bytes for the headline metric, though capturing the other fields is cheap and often useful for debugging. A minimal reader walks /proc, parses each numeric PID, opens the io file, and stores values in a dictionary keyed by PID.

Processes routinely exit between iterations, so wrapping the open call in a try/except block that catches FileNotFoundError and OSError keeps the daemon robust. Persisting the snapshot to disk lets longer deployments survive restarts without producing misleading spikes caused by a missing baseline.

Handling permissions and access control

On a hardened host, ordinary users can only inspect their own processes, so the script needs to run as root or under a dedicated service account. A common pattern in Australian government and academic environments, following guidance from the Australian Cyber Security Centre, is to confine the monitor to a least-privilege user that owns the binary and has setuid root or a carefully scoped sudoers entry.

Production deployments should drop privileges after binding to a fixed port if the script exposes a web interface. systemd's DynamicUser feature handles UID allocation and cleanup automatically, and a configurable log path keeps the tool portable across the Debian-derived hosts common in Australian hosting and the RHEL-based distros favoured by enterprise finance workloads.

Calculating throughput and presenting results

With raw counters captured at two timestamps, the read rate is (r1 - r0) / (t1 - t0). Multiplying by 1024 or 1000 depending on whether the consumer prefers KiB/s or KB/s is a small but appreciated touch. Colour-coded output via ANSI escapes helps when running interactively in a terminal in a Brisbane co-working space, while plain CSV works better when piping into Grafana or a cron-driven report. A flexible design offers both modes behind a --format flag.

A curses-style full-screen view that redraws in place is far easier to read than scrolling output, and the standard library's curses module handles this without dependencies. Sorting displayed rows by current throughput makes hot processes jump to the top, and a one-second refresh interval strikes a reasonable balance between responsiveness and overhead.

Aggregation, filtering and container awareness

A flat list of every PID can overwhelm a busy host running hundreds of containers. Filtering by command name, regex, or parent PID is a common requirement, and walking /proc/[pid]/cmdline and /proc/[pid]/status to read comm and PPid gives enough context to build filters without external libraries.

Inside a container, /proc only shows that container's processes, which is usually what you want. On the host, however, the same script can be combined with cgroup paths to attribute I/O to specific pods or systemd services. For Kubernetes clusters hosted on AWS Sydney or Azure Australia Central, this pattern integrates neatly with Prometheus exporters. Grouping by cgroup or executable basename is often more informative than per-PID reporting because forks and worker pools would otherwise spam the table.

Alerting and threshold triggers

A passive viewer is useful during debugging, but production setups usually want the script to raise an alarm when something abnormal happens. A simple approach is to compare each process's write_bytes delta against a configurable threshold and print a warning to stderr, log to syslog, or POST to a webhook used by the on-call rotation. Thresholds in the tens of megabytes per second catch runaway log shippers and broken backup jobs without firing on routine activity.

The Australian Signals Directorate's Essential Eight, frequently cited by mid-sized Australian businesses, encourages logging and alerting on anomalous resource use. A monitor that writes structured JSON to a file feeds easily into Filebeat, Vector, or a SIEM. For home users on the NBN, the same threshold logic warns when a torrent client or backup sync saturates the local SSD.

Running the script as a long-lived service

Production monitoring rarely lives in a terminal. Wrapping the script with a simple loop, a SIGTERM handler, and a lock file turns it into something safe to start under systemd or supervisor. Saving the previous sample to a small SQLite database allows the monitor to resume after a restart without losing the baseline, and periodic snapshots make it possible to reconstruct after-the-fact timelines during post-incident reviews.

Distributing the tool as a single-file script with a permissive licence keeps adoption friction low. Australian open-source maintainers frequently present lightweight monitoring tools at Linux Conference Australia and PyCon AU, where discussions surface refinements such as adding eBPF probes or exporting to OpenTelemetry. A small README documenting polling interval, threshold defaults, and limitations on older kernels rounds out a dependable diagnostic building block.