Building a Scheduled Python Script for SSH Key Rotation

SSH keys provide strong, convenient authentication for Linux servers, but long-lived credentials can become a serious security liability. Team members change roles, contractors finish assignments, and private keys may be copied to laptops that are later lost or replaced. A planned rotation process reduces the time available for an exposed key to be abused.

Python is well suited to this task because it can read configuration files, create key pairs, call system utilities, and record each operation in an audit log. A small script can rotate keys on a single VPS, while a carefully designed version can coordinate changes across a fleet hosted in Sydney, Melbourne, or an Australian cloud region.

The safest approach is staged replacement rather than immediate deletion. The new public key should be installed and tested before the old key is removed. Scheduling, backups, permissions, and rollback procedures matter as much as the key-generation command itself.

Define The Rotation Policy

Start by deciding what “rotation” means for the environment. A personal development server might replace a key every 90 days, while a production platform handling customer data may use a shorter interval. The policy should identify key owners, systems covered, approved algorithms, retention periods, and the person responsible for responding to failures.

Use separate keys for separate purposes. An administrator’s laptop key should not also authenticate automated deployments, backups, and monitoring services. Distinct identities make revocation precise and help reveal unusual access in audit records. For most current OpenSSH installations, Ed25519 keys are a practical default because they are compact and fast.

The script should also define a grace period. Keeping the previous key active for a few hours or a day allows operators to test the new credential and recover from a missed deployment. After that period, the old public key can be removed from authorized_keys and securely retired.

Generate Keys Without Exposing Secrets

Python can invoke ssh-keygen, although a well-maintained cryptographic library may be preferable when the program needs to manage keys directly. A simple subprocess call should use an argument list rather than a shell command assembled from untrusted strings.

import subprocess
from pathlib import Path

key_path = Path("/var/lib/ssh-rotation/service_ed25519")

subprocess.run(
    [
        "ssh-keygen",
        "-t", "ed25519",
        "-f", str(key_path),
        "-N", "",
        "-C", "service-rotated",
    ],
    check=True,
)

In production, avoid an empty passphrase unless the key is restricted to a tightly controlled service account and protected by another mechanism. A passphrase can be supplied through a secret manager, though unattended jobs then need an agent or secure retrieval process. Set the private key mode to 0600, keep the containing directory inaccessible to other users, and never print private key content to standard output.

Install The New Public Key Safely

A rotation job needs a controlled method for installing the new public key. For a remote host, the script can use an existing administrative key with ssh, or it can call a configuration-management platform. The deployment account should have the minimum permissions required to update the target account’s authorized_keys.

Avoid rewriting the entire file unless the script owns every entry. Existing keys may belong to another administrator, a monitoring system, or an emergency access account. Instead, add a clearly labelled line and maintain a record of the key fingerprint. A marker such as managed-by-rotation:2026-08 makes later removal safer.

Before changing anything, copy the existing file to a root-readable backup with restrictive permissions. Validate that the resulting file has the expected owner, mode, and syntax. An accidental 0777 permission or incorrect ownership can cause OpenSSH to reject authentication and create an avoidable outage.

Verify Access Before Revocation

The strongest safeguard is an independent login test using the new private key. Python can run an ssh command with a short connection timeout, disable interactive prompts, and request a harmless command such as id. The test should confirm both successful authentication and the expected remote account.

Do not test from the same session that could be carrying an existing connection. A script may report success because the SSH agent reused an old credential. Use a fresh process, explicitly select the new identity with -i, and set IdentitiesOnly=yes. If the destination is behind a bastion host, test the full ProxyJump path used by the real service.

Only after verification should the old key enter its grace period. A failed test must stop the removal phase, send an alert, and preserve the prior access path. This staged sequence is especially valuable for small teams in Brisbane or Perth, where an overnight failure may otherwise wait until the next business day.

Schedule The Job With Clear State

A Linux systemd timer is often more reliable than a long-running Python loop. The service can run with a dedicated account, a fixed working directory, restricted environment variables, and resource limits. A timer with a randomised delay prevents many hosts from contacting a central service at exactly the same moment.

For smaller systems, cron remains suitable. Use an absolute path to the Python interpreter and script, redirect output to a controlled log destination, and make the job’s timezone explicit. Australian organisations should account for daylight saving differences between states: Sydney and Melbourne change clocks, while Brisbane does not. Scheduling in UTC avoids ambiguous local times.

The program should store state outside its source directory. A small JSON or SQLite record can hold the current fingerprint, creation timestamp, deployment targets, and rotation phase. Atomic writes, such as writing a temporary file and renaming it, prevent a power interruption from leaving incomplete state.

Handle Failures And Rollback

Network timeouts, DNS problems, disk exhaustion, and a locked account can interrupt rotation. Catch expected exceptions and classify them clearly rather than returning a generic failure. A retry policy with exponential backoff is useful for transient network faults, while authentication failures should normally stop immediately and trigger investigation.

Rollback should restore the previous authorized_keys file or remove only the newly added entry. Preserve the old private key in a protected location until every target has passed verification and the grace period is complete. Once it is no longer required, securely delete it according to the organisation’s data-handling policy.

A deployment should also have an emergency access method that does not depend on the rotating credential. Console access from a cloud provider, a hardware-backed administrator key, or a documented break-glass account can prevent a routine security task from becoming a lockout event.

Record Auditable Results

Each run should record the time, host, account, old and new fingerprints, outcome, and error category. Never log private key material, passphrases, secret-manager responses, or complete authentication commands containing sensitive values. Logs should be writable by the service but readable only by authorised operators.

Centralise useful events in the same monitoring system used for other infrastructure. A sudden series of failed rotations can indicate a configuration error or an active attack. Log retention should match internal policy and any contractual requirements affecting Australian customers or regulated workloads.

Web and access logs can provide useful context when a server is being investigated. For example, Apache log analysis can help distinguish routine crawler activity from suspicious automation around the time of an SSH alert. The systems should remain separate, but correlating timestamps and source addresses may reveal a broader incident.

Test The Rotation Before Production

Build a disposable test environment that mirrors production account names, SSH configuration, permissions, bastion routing, and scheduling. Run the script repeatedly, including interrupted runs and partially completed deployments. Confirm that rerunning it is safe and does not create duplicate keys or delete unmanaged access.

Test clock boundaries, daylight saving transitions, expired credentials, unavailable hosts, and simultaneous jobs. Verify that alerts reach the right people and that a failed rotation leaves a usable previous credential. A dry-run mode can show planned changes without modifying remote files, but it must be clearly labelled and must not be mistaken for verification.

Once the process is dependable, deploy it gradually. Start with a non-critical host, review its audit records, then expand in batches. Regularly review key fingerprints and remove obsolete accounts so that automated rotation strengthens the access model instead of preserving unnecessary permissions.