Building a reliable Python worker daemon

A long-running service often has more work to process than one Python process can handle comfortably. A pool of reusable workers spreads CPU-heavy tasks across separate processes while avoiding the startup cost of launching a new interpreter for every job.

As a senior software developer, Phil Schwartz has built focused open-source tools around practical problems. DenyHosts watches SSH login failures, Kodos inspects regular expressions, and Scratchy turns Apache logs into useful information. Their shared strength is a narrow purpose rather than a sprawling framework.

On Linux, the best “daemon” is often an ordinary foreground service placed under systemd. A supervising parent can distribute work, detect failed children and shut everything down cleanly, giving the pool a predictable lifecycle.

Choose the right concurrency model

Use processes for CPU-bound work and for jobs that need isolation. Threads remain appropriate for lightweight network waits, while a process pool gives each worker its own interpreter and address space. For straightforward parallel jobs, concurrent.futures.ProcessPoolExecutor handles much of the plumbing; a custom pool earns its keep when persistent state, explicit deadlines or rolling worker replacement matter.

Choose the multiprocessing start method deliberately. Fork is inexpensive on Linux, but spawn is predictable across platforms and avoids inheriting a complicated parent state. Under spawn, worker targets must be importable module-level functions, and the entry point must be protected by if __name__ == '__main__':.

Avoid designing the pool as an extension of multiprocessing.Pool without checking what you need. A custom manager can expose worker heartbeats, job limits and clear shutdown states, yet it also introduces queue, signal and restart logic that must be tested.

Design tasks as a small protocol

Give the manager and workers a small, versioned message protocol. A request should carry a request ID, an operation name and validated arguments. Dispatch through an allowlist of module-level functions rather than accepting arbitrary code or executable payloads. Replies should separate successful output from sanitised errors and include the worker PID and elapsed time.

Bound the request and result queues so a traffic burst cannot consume all available memory. Apply backpressure when the request queue is full, and record rejected work instead of silently dropping it. For operational visibility, expose queue depth, worker age and recent completion rate through a status endpoint or command-line tool.

Use time.monotonic() for deadlines, not wall-clock time, because clock adjustments can invalidate a timeout. If a job overruns its deadline, the manager should stop and replace that worker rather than return it to the pool with its state unknown. Recycling workers after a fixed task count also limits memory growth from third-party libraries.

Supervise workers and service lifecycle

The supervisor should treat a worker exit as an event, not a silent capacity loss. Track heartbeats and child PIDs, restart dead workers with capped backoff, and stop after repeated failures if the underlying service is unavailable. A restart limit prevents a crash loop from becoming a fork bomb.

Handle SIGTERM by stopping new requests, allowing active work to finish, then joining workers and closing queues. Give the drain period a deadline and escalate to SIGKILL only when necessary. Avoid multiprocessing’s daemon worker flag for jobs that need cleanup, because those processes are terminated abruptly.

On modern Linux, let systemd own the service lifecycle instead of using a traditional double-fork. A unit can specify the virtual-environment interpreter, an unprivileged User, Restart=on-failure, RuntimeDirectory and a stop timeout. Absolute executable paths and a defined WorkingDirectory also remove startup surprises.

Deploy securely for Australian operations

Australian deployments range from a home Linux box in Brisbane on an NBN connection, which may use fibre or fixed wireless, to a managed server for a Sydney or Melbourne business. These environments often sit behind consumer routers or cloud firewalls. Keep the daemon on a real Linux host, restrict listening ports and avoid assuming that a dynamic home address will remain stable. Keep timestamps in UTC internally and show the relevant Australian time zone in reports, including daylight-saving changes.

Logs are operational data and sometimes personal data. Names, IP addresses and request details can fall within the federal Privacy Act 1988, depending on the organisation and information involved. Australian organisations covered by the Act should also have an incident process for the Notifiable Data Breaches scheme, including assessment of whether notification is required. Minimise log content and set retention periods rather than keeping every raw record forever.

A companion email notifier must also comply with the Spam Act 2003, including consent, sender identification and a working unsubscribe mechanism where required. The Australian Signals Directorate’s Essential Eight is a widely used security benchmark among Australian government agencies and businesses, though it is not a general law for every organisation. It remains a useful reference for access controls, patching and incident preparation.

Practical recommendations for production use

A production-oriented Python service benefits from the same simplicity seen in small open-source utilities: a narrow purpose, a readable dependency set and an installation path that does not require guesswork.

• Run the service under systemd rather than an improvised shell loop, and exercise a complete restart during testing.
• Keep one request type per worker function so serialisation boundaries and failure modes stay obvious.
• Add an idempotency key to jobs that may be retried, particularly anything that writes files or sends notifications.
• Test fork and spawn modes, worker crashes, slow tasks, queue saturation and shutdown during the busiest period.
• Record structured metrics for queue depth, job duration, worker restarts and failed tasks without logging sensitive payloads.
• Publish the licence and dependency obligations clearly; a copyleft component can affect how a modified package is distributed.
• Document the supported Python versions, configuration format and upgrade procedure on the project page.

This approach suits Phil Schwartz’s portfolio because it turns reusable process management into a dependable foundation for small, specialised tools. The resulting services can remain readable, observable and easy to operate without presenting themselves as full-scale frameworks.