Building a python circular buffer for in-memory log rolling

Software that runs continuously eventually fills its disk with log lines. For services deployed across Australian data centres in Sydney, Melbourne, and beyond, the problem becomes more pronounced because retention rules under the Privacy Act 1988 push operators toward keeping detailed audit trails. A circular buffer offers an elegant compromise: keep the most recent activity in memory, drop older entries automatically, and flush snapshots to disk on a schedule.

Python's standard library provides everything required to build such a structure without external dependencies. The combination of collections.deque, threading.Lock, and a few helper methods gives a developer a robust mechanism that survives bursts of traffic and midnight cron jobs alike.

For teams hosting workloads in regions governed by the Australian Cyber Security Centre's Essential Eight, predictable memory ceilings also simplify compliance reporting. Instead of arguing about unbounded growth, you can point to a known upper bound measured in megabytes per process.

This article walks through a practical implementation. The aim is a self-contained module that other open-source utilities, similar to DenyHosts or Scratchy, can drop into a project and rely on.

Why a circular buffer matches log rotation

A circular buffer, also known as a ring buffer, behaves like a queue with a fixed capacity. Once the buffer is full, new entries overwrite the oldest ones. That semantic matches the real-world need of an operator who cares about the last hour of activity far more than the last month.

In contrast to file-based rotation handled by logrotate, an in-memory ring keeps latency low. Writes happen at memory speed, and reads for inspection happen without touching disk. For high-frequency events such as SSH connection attempts or Apache access lines that a tool like Scratchy might process, this matters.

The fixed size also removes a class of bug. There is no scenario where a runaway process writes gigabytes of traceback into a file and exhausts the filesystem. The buffer simply evicts, and the eviction policy is decided in code rather than by whichever cron job happens to fire next.

Selecting the underlying data structure

Python ships with collections.deque, which supports O(1) appends and pops from either end. Setting maxlen turns it into a ring buffer out of the box. The downside is that deque does not expose indexed access as cleanly as a NumPy array would, but for log payloads that trade-off is acceptable.

If the workload demands structured rows with timestamps, severities, and message bodies, a list of tuples wrapped around a length check can be clearer. Each entry stays a small dataclass, and the overwriting logic lives in a single append method. That pattern reads well in code review and aligns with the style encouraged at Australian Python community meetups such as the Sydney and Melbourne PyCon gatherings.

For very high write rates, the array module or a NumPy ndarray can hold primitive timestamps more compactly. Most log-handling code rarely needs that level of optimisation, and keeping the implementation in pure Python makes it portable across the Linux distributions common on Australian servers.

Constructing the buffer class

The skeleton begins with a constructor that accepts the maximum number of entries and an optional capacity measured in characters. A threading.Lock guards mutations, and a private deque holds the records.

from collections import deque
from threading import Lock
from dataclasses import dataclass
from time import time

@dataclass
class LogEntry:
    ts: float
    level: str
    message: str

class RingLog:
    def __init__(self, capacity=10000):
        self._buf = deque(maxlen=capacity)
        self._lock = Lock()

The append method acquires the lock, formats the entry, and pushes it onto the deque. The maxlen argument handles the overwriting automatically, so the method stays short. Reading methods return a snapshot list under the same lock to keep iterators from seeing a partially mutated structure.

Handling concurrency safely

A web-facing daemon will have multiple threads writing log lines. Without synchronisation, two threads could read the same slot before either writes, producing torn records. The lock pattern in the skeleton above is enough for moderate load.

For multi-process setups, such as a Gunicorn worker pool behind nginx on a VPS in Brisbane or Perth, the in-memory buffer should be per-process. Coordination between workers is the job of an external aggregator like syslog or a message queue. Trying to share one ring across processes introduces shared-memory complexity that defeats the purpose of a simple solution.

A useful idiom is to expose a snapshot() method that copies the buffer under the lock and returns it. Downstream consumers, including the disk-flush routine discussed below, operate on the snapshot without blocking writers.

Implementing time-based eviction

Capacity is one eviction policy; time is another. A ring that holds 10,000 entries is meaningless if those entries span a week of quiet traffic followed by a one-second spike. To bound the window in seconds, each entry stores its timestamp and a sweep removes anything older than the configured horizon.

A helper method iterates from the left of the deque while the oldest entry is stale, popping it. The amortised cost is low because most calls do nothing, and a burst that genuinely fills the buffer pays a one-time cost per old entry. Australian operators running services during AEDT business hours will appreciate the timestamp being stored as a Unix epoch and formatted in local time on demand.

Flushing to disk without blocking writes

Periodic persistence protects against process crashes. A background thread, started with threading.Thread(daemon=True), wakes every N seconds, takes a snapshot, and writes it to a file. The snapshot is independent of the live buffer, so producers are not blocked.

Rotation of the file itself can follow a simple naming scheme such as service.log, service.log.1, and so on. Shifting the files is the same os.rename dance that classic log rotation uses, scaled down to the in-memory history.

For Australian privacy considerations, the flush routine should respect any configured redaction. Storing passwords or session tokens in the buffer is rarely intentional, and a redaction hook called before append keeps the buffer honest.

Validating behaviour under load

A short unit test that pushes more entries than the capacity and asserts that len(buffer) == capacity catches the most common regression. A second test verifies that the oldest entries are the ones dropped.

For higher confidence, a property-based test using Hypothesis generates random bursts and confirms that the buffer never exceeds its capacity and that every retained entry was actually appended. Running these tests inside a CI pipeline hosted in Sydney or another Australian region keeps round-trip latency low for local developers and satisfies data-residency preferences common in government-adjacent projects.

Recommendations for production use

• Pick a capacity that fits roughly five minutes of peak traffic to bound memory growth without losing useful context.
• Store timestamps as Unix epochs and convert to local AEST or AEDT only when formatting for display.
• Wrap the buffer with a redaction hook so secrets never reach memory in the first place.
• Use a daemon thread for flushing, and keep the flush interval shorter than your retention window.
• Prefer a per-process buffer and rely on syslog or a sidecar for cross-process aggregation.