Build a Python systemd journal exporter

systemd’s journal gives Linux services a single, queryable record of their output. A Python daemon can follow that stream and copy selected events into a custom file for auditing, search or long-term retention. This works well when shell redirection is too limited or downstream tools need consistent JSON.

The exporter described here uses journalctl to receive journal events as JSON, then writes JSON Lines to a dedicated file. It relies primarily on the Python standard library, making it suitable for servers, cloud instances and locally managed Linux systems. Python 3.9 or later is recommended.

Like the open-source utilities in Phil Schwartz’s portfolio, this approach favours readable source code and familiar Linux facilities. The daemon runs under systemd, receives only the privileges it needs and remains separate from the services being observed. Its configuration is small enough to audit without a framework or application server.

Because journal messages can contain credentials, customer identifiers or other sensitive information, the destination file needs restrictive permissions. Treat the exporter as part of the host’s security boundary, and avoid forwarding its contents to third-party monitoring services unless the data-handling terms are appropriate.

Understand the data flow

The journalctl --follow --output=json command emits each journal entry as a JSON object. Useful fields include MESSAGE, __REALTIME_TIMESTAMP, __CURSOR, _SYSTEMD_UNIT, SYSLOG_IDENTIFIER and _HOSTNAME. The cursor identifies the entry’s position in the journal and can support recovery after an interruption.

Each output object becomes one line in the custom file. This JSON Lines format remains readable with ordinary text tools while preserving structured fields for jq, Python or log-processing systems. Timestamps are converted from UTC into a configurable Australian time zone, while the original cursor and host remain available for correlation.

Prepare the logging daemon

Save the following as journal_exporter.py. It opens the output file once, follows journal events and restarts journalctl after an unexpected disconnection:

#!/usr/bin/env python3
import json
import os
import signal
import subprocess
import time
from datetime import datetime, timezone
from pathlib import Path
from zoneinfo import ZoneInfo

OUTPUT = Path(os.environ.get(
    "OUTPUT_FILE", "/var/log/journal-export/events.jsonl"
))
TIMEZONE = ZoneInfo(os.environ.get(
    "LOG_TIMEZONE", "Australia/Sydney"
))

running = True
child = None

def stop(signum, frame):
    global running, child
    running = False
    if child is not None and child.poll() is None:
        child.terminate()

def write_record(source):
    stamp = int(source.get("__REALTIME_TIMESTAMP", 0))
    when = datetime.fromtimestamp(
        stamp / 1_000_000, timezone.utc
    ).astimezone(TIMEZONE)

    return {
        "time": when.isoformat(timespec="milliseconds"),
        "host": source.get("_HOSTNAME", ""),
        "unit": source.get("_SYSTEMD_UNIT")
            or source.get("SYSLOG_IDENTIFIER", "-"),
        "message": source.get("MESSAGE", ""),
        "cursor": source.get("__CURSOR", ""),
    }

signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)

with OUTPUT.open("a", encoding="utf-8") as output:
    while running:
        process = subprocess.Popen(
            ["journalctl", "--follow", "--output=json", "--no-pager"],
            stdout=subprocess.PIPE,
            text=True,
            bufsize=1,
        )
        child = process

        try:
            for line in process.stdout:
                if not running:
                    break
                try:
                    source = json.loads(line)
                except json.JSONDecodeError:
                    continue

                record = write_record(source)
                output.write(
                    json.dumps(record, ensure_ascii=False) + "\n"
                )
                output.flush()
        finally:
            child = None
            if process.poll() is None:
                process.terminate()
                try:
                    process.wait(timeout=5)
                except subprocess.TimeoutExpired:
                    process.kill()
                    process.wait()

        if running:
            time.sleep(2)

The script deliberately uses no non-standard Python packages. Its SIGTERM handler also stops the journalctl child, allowing systemd to shut the service down cleanly rather than leaving a background process behind.

Configure the systemd service

Install the script and create a dedicated unprivileged account. The systemd-journal group grants the required journal access on distributions that use the standard group arrangement:

sudo install -d -m 0755 /usr/local/bin
sudo install -m 0755 journal_exporter.py /usr/local/bin/journal-exporter
sudo useradd --system --user-group --home /nonexistent \
  --shell /usr/sbin/nologin journal-exporter
sudo usermod -a -G systemd-journal journal-exporter

Create /etc/systemd/system/journal-exporter.service with this configuration:

[Unit]
Description=Export systemd journal entries to JSON Lines
After=systemd-journald.service

[Service]
Type=simple
User=journal-exporter
Group=journal-exporter
SupplementaryGroups=systemd-journal
Environment=LOG_TIMEZONE=Australia/Sydney
Environment=OUTPUT_FILE=/var/log/journal-export/events.jsonl
ExecStart=/usr/local/bin/journal-exporter
Restart=on-failure
RestartSec=3
UMask=0077
LogsDirectory=journal-export
StateDirectory=journal-exporter
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
LockPersonality=yes

[Install]
WantedBy=multi-user.target

LogsDirectory creates /var/log/journal-export with the correct ownership, while StateDirectory provides a protected location for a persistent cursor. The restrictive UMask prevents other local users from reading the exported records. If the systemd-journal group is absent, consult the access rules used by the host distribution before changing permissions.

Apply Australian operating details

The configured Australia/Sydney zone handles AEST and AEDT daylight-saving transitions for Sydney and Melbourne. Brisbane remains on AEST, while Perth uses AWST. A team can select Australia/Brisbane, Australia/Perth or another zone through LOG_TIMEZONE, and the resulting ISO 8601 timestamps retain the correct UTC offset.

For Australian managed-hosting and cloud customers, local support teams often need timestamps that match business hours in Sydney and Melbourne. This makes incident timelines easier to interpret during daylight-saving changes and reduces confusion when comparing exporter records with Australian operational dashboards.

Runbooks should use Australian English conventions such as “behaviour” and “licence”, although the text inside third-party MESSAGE fields must remain unchanged. For organisations supporting sites in Australia and Japan, the on-call runbook can pair service locations with a Japanese map portal, while all journal processing remains on the local server.

Add rotation and recovery

A tail-style feed can leave a gap if the exporter stops while the journal continues to receive events. For dependable recovery, save each __CURSOR in the state directory and start journalctl with --after-cursor=VALUE. Writing the exported event before advancing the cursor provides at-least-once delivery, although consumers may need to remove duplicates after a crash.

The custom file also needs independent rotation. This logrotate configuration retains two weeks of compressed copies:

/var/log/journal-export/events.jsonl {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Because the Python process holds the file open, copytruncate is the simplest option, with a very small race during truncation. A more elaborate exporter can handle SIGHUP and reopen the destination. Journald’s own retention is controlled separately through SystemMaxUse and related settings in journald.conf.

Verify and maintain the exporter

Load the unit, start it and create a test event:

sudo systemctl daemon-reload
sudo systemctl enable --now journal-exporter.service
sudo logger "journal exporter test"
sudo tail -f /var/log/journal-export/events.jsonl

Each visible line should be a complete JSON object containing the local time, hostname, unit, message and cursor. Service errors appear in the exporter’s own journal and can be inspected with sudo journalctl -u journal-exporter -n 50. Restarting the service verifies that systemd supervision and file ownership behave as expected.

For an open-source release, publish the script, unit file, supported distribution details, security assumptions and licence together. Clear documentation and a repeatable installation path turn the exporter into a practical Linux utility that operators can audit, deploy and maintain confidently.