Creating a Python Tool to Detect SMTP Open Relay Misconfiguration

Open relay misconfigurations remain one of the oldest and most exploited weaknesses in email infrastructure. An SMTP server that relays mail for untrusted senders becomes a free launchpad for spammers, and the fallout often lands on the operator. In Australia, the Australian Communications and Media Authority monitors abuse complaints, and misconfigured relays hosted on local networks can attract rapid attention from upstream providers like Telstra and Optus.

Many administrators rely on remote testing portals, but those services can be slow, rate-limited, or unavailable during an incident. Writing a self-contained Python script gives you a private, repeatable audit tool that you can run from a laptop in Sydney, Melbourne, or a regional office in Hobart. This approach mirrors the philosophy behind utility projects hosted on personal developer pages such as phil-schwartz.com, where lean, single-purpose scripts solve real operational headaches.

The task combines socket programming, DNS lookups, and the standard smtplib module. By the end of a working build, your tool will probe a target mail exchange, attempt to relay a probe message without credentials, and flag any server that accepts the transaction.

Understanding open relay vulnerabilities in SMTP

An open relay accepts and delivers messages from third parties who neither authenticate as legitimate users nor originate from trusted IP ranges. The classic scenario involves the SMTP commands MAIL FROM and RCPT TO, where the server does not validate that the sender has permission to use it as a transport. Early Sendmail deployments, in particular, were notorious for this default behaviour.

Modern mail transfer agents like Postfix, Exim, and Microsoft Exchange ship with safer defaults, yet misconfigurations persist. A common mistake involves allowing mynetworks to expand too broadly, or leaving a backup smarthost with permissive relay rules during a migration. Attackers scan port 25 relentlessly, and an exposed relay can be blacklisted within hours, harming deliverability for every customer of the affected Australian hosting provider.

Testing for the flaw involves sending a probe message with an arbitrary sender address to an arbitrary recipient on a different domain. If the server returns 250 OK for both RCPT commands and delivers the test, it relays for the open internet. This canonical check is what any custom checker must reproduce.

Setting up your Python development environment

Python's standard library already contains everything required for the job. The smtplib module handles SMTP conversations, while socket and the dns libraries manage the underlying transport. If you prefer a third-party resolver, the dnspython package offers a clean wrapper that handles MX queries without manual UDP gymnastics.

A virtual environment keeps the project tidy and reproducible. Create a new directory, run python3 -m venv venv, then activate it before installing any dependencies. Most Australian developers working on Linux servers over the National Broadband Network will find that pip fetches packages without issue, although corporate firewalls occasionally block PyPI mirrors.

Plan for logging early, because relay testing will generate noise you need to triage. The logging module supports rotating files out of the box, and combining it with structured output makes it easier to feed results into a SIEM or a simple dashboard later.

Parsing MX records to find the right target

Before you can probe an SMTP server, you need to know where to send the connection. Mail for a domain is handled by the hosts listed in its MX records, ordered by preference. Resolving those records manually means issuing a DNS query of type MX and sorting the responses by their priority values.

A compact helper using dnspython looks like this:

import dns.resolver

def get_mx_hosts(domain):
    answers = dns.resolver.resolve(domain, 'MX')
    return sorted(
        [(r.preference, str(r.exchange).rstrip('.')) for r in answers]
    )

Some domains publish no MX record at all, in which case the A record serves as a fallback. Handle both cases gracefully, since Australian government agencies and small businesses occasionally operate without explicit MX entries during legacy migrations.

Crafting the relay test logic with smtplib

With the target host in hand, the next step is the SMTP conversation itself. Open a connection on port 25, read the banner, and issue EHLO with your hostname. The script then sends MAIL FROM with an obviously external address such as probe@example.com, followed by RCPT TO targeting a recipient on a domain you control. Capture every response code along the way.

A clean open relay responds with 250 to both RCPT commands and accepts the message body. If the server requires authentication and challenges with 530, or rejects the recipient with 553, the relay is closed. Anything in between warrants manual review, since some servers will relay after STARTTLS negotiation only when the connection originates from a whitelisted IP.

Throttle your requests. Bombarding a target with rapid connections will trip intrusion detection systems and could get your own IP listed by Australian ISPs running outbound SMTP filters. A simple time.sleep between attempts keeps the audit polite.

Handling edge cases and reducing false positives

Greylisting, temporary failures, and DNS-based reputation services can muddy results. A server that returns 450 on the first attempt might accept the relay a minute later. Your checker should retry transient failures a configurable number of times before logging a positive.

Some organisations use a submission port (587) for authenticated relay and block port 25 entirely from the public internet. Always document which port you tested and the banner you received, since administrators reviewing the report will want that context.

Build a small allowlist of your own domains so the script does not flag your own infrastructure. Many Australian sysadmins keep the test recipient on a subdomain such as relaycheck.example.com.au to keep probe traffic isolated from production mailboxes.

Hardening mail servers after the audit finishes

If your script reports an open relay, the fix usually lives in the MTA configuration. For Postfix, tightening mynetworks to include only localhost and your internal subnets closes the loophole immediately. Sendmail users should review the DnR and R= rules in the mailertable.

Enforce SMTP authentication using SASL, and require STARTTLS on every outbound connection. Publish SPF, DKIM, and DMARC records so receiving servers can validate your mail. Australian senders that fail to implement these protections often see their messages rejected by major recipients such as Gmail and Outlook, regardless of how clean the relay posture is.

Automate the check. Run the script weekly from a cron job and email yourself a summary. Pairing automated auditing with manual spot checks keeps small mistakes from becoming large incidents.

Common SMTP response codes worth watching

• 250: command accepted; a likely sign of an open relay when received after MAIL FROM and RCPT TO from external addresses
• 450: temporary failure; often part of greylisting, retry before declaring a result
• 530: authentication required; the relay is correctly closed
• 550: permanent failure; recipient or sender rejected by policy

Practical hardening steps for Postfix relay servers

• Restrict mynetworks to localhost and your trusted internal subnets only
• Enable smtpd_sasl_authenticated_header so authenticated clients bypass relay checks
• Require smtpd_tls_security_level = encrypt to mandate STARTTLS on inbound sessions
• Publish SPF, DKIM, and DMARC records so receiving servers can validate your traffic